Performance Alerts

Monitoring ML model performance on Qwak is essential for maintaining the reliability and effectiveness of your models. Our integrated alerting system enables you to effortlessly track key metrics and receive realtime notifications. It seamlessly integrates with your preferred communication and incident management tools, such as Slack or OpsGenie.

🚧

Note: Alerts are available for Hybrid clients only and are not included in the Qwak Cloud offering.

Configuring Alerts

1. Open the model you'd like to monitor and switch to the Alerts tab.
2. Click Add Alert.
3. In the Alerts table, choose the required metric type, aggregation, and condition. In the period column, specify the time window (in minutes).
4. From the Channels dropdown, select the Slack channel to receive the notifications. If you don't see your channel there, follow the instructions below to add a new channel.
5. Pick which model variant should be tracked or choose "All variations".
6. Remember to save your enable the alert by clicking the Status toggle and save the changes using the Save button!
7. For deployments of type Streaming the only supported alert types are "Error rate" and "Throughput". All alerts of type "Latency" will be automatically disabled. Remember to re-enable those alerts in case you change your deployment time to Realtime

Configuring Channels via the UI

Configure channels via: Account Settings -> Notifications -> Add channels

📘

Note: Channel are configured per environment. For accounts that support multiple environments, please configure channels per environment.

Configuring Slack Notifications

Incoming Webhooks are a simple way to post messages from external sources into Slack. They make use of normal HTTP requests with a JSON payload, which includes the message and a few other optional details. You can include message attachments to display richly-formatted messages.

Adding incoming Webhooks requires a bot user. If your app doesn't have a bot user, one will be added for you. Each time your app is installed, a new Webhook URL is generated.

If you deactivate incoming Webhooks, new Webhook URLs will not be generated when your app is installed to your team. If you'd like to remove access to existing Webhook URLs, you will need to Revoke All OAuth Tokens.

1. Create a Slack channel for your notifications
2. Create a Slack application using https://api.slack.com/apps?new_app=1
3. Create an incoming Webhook:
   1. Navigate to Features > Incoming Webhooks
   2. Click Activate and Add New Webhook to Workspace
   3. Copy the Webhook.

Configuring Opsgenie Notifications

To setup Opsgenie you will need an API Key and the Alert API Url. These can be obtained by configuring a New OpsGenie Integration

1. Add Grafana Integration in Opsgenie
2. You can add this integration from your team dashboard
   1. Go to Settings -> Intgreation -> choose Grafana integration

Save the integration: