Getting Grafana Alerts Right: Provisioning on Kubernetes and Notifying Slack
Level up your Grafana Alerting setup by provisioning it on Kubernetes and sending notifications to Slack
Together with Prometheus and Loki, Grafana forms a complete and broadly used open-source stack for monitoring and observability. As powerful as these tools are, configuring and running them takes skill. And no matter how sophisticated your monitoring and observability setup is, it is of little use without carefully tuned alerts and notifications.
Why have a monitoring system at all if you cannot respond in time when your application goes down?
Starting with the Grafana 9.1 release, the whole Grafana Alerting stack can be provisioned as code, either with Kubernetes or with Terraform.
In this tutorial, I will show you how to set up Grafana alerts triggered by particular keywords in logs, and how to deliver the resulting notifications to a Slack channel using file provisioning in Kubernetes. Let’s dive in.
Prerequisites
Install Grafana with the kube-prometheus-stack Helm Chart. Deploy Loki into your cluster with the loki-stack Helm Chart.
Enable file provisioning for Grafana Alerting
In Kubernetes, file provisioning relies on ConfigMaps. The Grafana Alerting sidecar container, which lives in the same pod as the Grafana container, watches the ConfigMaps. Whenever it finds a ConfigMap labelled grafana_alert, it loads the provisioned resources into Grafana.
Below you can see what such a ConfigMap looks like. The templates of the Grafana Alerting resources go under the data field.
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-alerting
namespace: monitoring
labels:
grafana_alert: "1"
data:
grafana-alert.yaml: |-
The Grafana Helm Chart ships with the Grafana Alerting sidecar container turned off. To switch it on, put the following into the values of your kube-prometheus-stack Helm Chart:
grafana:
sidecar:
alerts:
label: grafana_alert
enabled: true
searchNamespace: ALL
labelValue: "1"
Deploy the Helm Chart and check that the alerting sidecar is running alongside Grafana.
Grafana Alerting resources
Here are the main building blocks of Grafana Alerting:
- alert rules - They specify the conditions under which an alert fires. Here, Loki is our data source, which lets us raise alerts when particular keywords such as
errororexceptionshow up in the logs. - contact point - It holds the configuration used to deliver notifications. In our setup, alert notifications are sent to Slack, which acts as the contact point.
- notification policy - With notification policies, you can group many alerts and direct them to different recipients. For instance, labels let you send several alerts to the same contact point.
- notification template - Notification templates let you build custom messages and reuse them across alert notifications.
Alert rules
The simplest way to provision a Grafana Alert is to build it in the Grafana UI and then export it to a YAML file.
To create an alert, open the Alerting view and go to the Alert Rules page. Press the Create Alert Rule button.
In Step 1, enter a name for your alarm.
In Step 2, pick Loki as the Datasource.
Our goal is to get an alert whenever the application throws an exception. Add the first expression, A, with the query below, which pulls the exception message out of the logs:
Create a query:
count_over_time({app="demo-app"} | json message="message" |~ `exception` [5m])
The count_over_time function counts the entries of every log stream within the specified range.
Next, add a second expression, B, of type Reduce, and a third expression, C, of type Threshold. Set expression B as the input and 0 as the is above value. With this setup, a notification is sent every time the exception keyword appears in the logs. Mark it as the alert condition.
In Step 3, select the folder in which the alert should be stored, along with the evaluation group. Set the evaluation period to 0 seconds so that the alert fires right away.
In step 4, fill in the alarm summary and, if you have one, the runbook URL. For the alarm description, use the message label extracted from the log. You can also point to the dashboard and panel linked to this alert rule – do this in the Link the dashboard and panel view.
In Step 5, add labels such as app=demo-app and environment=production. Labels make it possible to send alert notifications to different contact points.
At this point, you can export the alert rule and put it into the template inside the ConfigMap.
Exporting Grafana alert rules:
Open https://grafana-url/api/v1/provisioning/alert-rules/{alert_rule_ID}/export and copy the rule.
You can find the alert rule ID in the alert rule details:
Once the alert rule is exported, remove it from the Grafana UI so that it can be provisioned as code. If you try to provision an alert template through code under the same name as an alert you created in the Grafana UI and did not delete, you will get an error.
Slack message template
Next, let’s build reusable templates for the Slack messages.
You need two templates: one for the body of the Slack message and one for its title.
Message template:
templates:
- orgID: 1
name: slack.message
template: |-
{{ define "slack.print_alert" -}}
*Alert*: {{ .Labels.alertname }}
{{ if .Annotations -}}
*Summary*: {{ .Annotations.summary}}
*Description*: {{ .Annotations.description }}
{{ end -}}
*Log message*: {{ index .Labels "message" }}
*Explore logs:* https://grafanaURL.com/explore?orgId=1
{{ if .DashboardURL -}}
*Go to dashboard:* {{ .DashboardURL }}
{{- end }}
{{ if .PanelURL -}}
*Go to panel:* {{ .PanelURL }}
{{- end }}
*Details:*
{{ range .Labels.SortedPairs -}}
- *{{ .Name }}:* `{{ .Value }}`
{{ end -}}
{{ if .SilenceURL -}}
*Silence this alert:* {{ .SilenceURL }}
{{- end }}
{{- end }}
{{ define "slack.message" -}}
{{ if .Alerts.Firing -}}
{{ len .Alerts.Firing }} firing alert(s):
{{ range .Alerts.Firing }}
{{ template "slack.print_alert" . }}
{{ end -}}
{{ end }}
{{- end }}
To make the Slack message template easier to follow, let’s go through its parts. At the top there are the alert name, the summary and the description. Right after them comes the log message extracted from the logs.
Then you’ll see a link to the Explore logs page in Grafana.
If the alert rule defines them, you will also be able to jump to the dashboards and panels connected to the alert.
All other labels are listed in the message to provide more details.
Finally, there is a link that lets you mute the alert quickly whenever you need to.
Title template
templates:
- orgID: 1
name: slack.title
template: |-
{{ define "slack.title" -}}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] Grafana Alerting Notification
{{- end -}}
Contact point
The Slack channel #grafana-alerts will serve as our contact point. To register Slack as a contact point, start by creating a basic Slack app and turning on incoming webhooks. Then set up an Incoming Webhook for your Slack channel. You’ll find the details in this guide. A sample configuration looks like this:
contactPoints:
- orgId: 1
name: grafana-alerts
receivers:
- uid: grafana
type: slack
settings:
username: grafana_bot
url: incoming_webhook_url
title: |
{{ template "slack.title" . }}
text: |
{{ template "slack.message" . }}
Notification policy
The root notification policy acts as the fallback receiver for alerts whose labels match nothing else. Point it at your default channel or configure Grafana email notifications.
Then add a child notification policy that sends every alert labelled app=demo-app to the #grafana-alerts Slack channel.
policies:
- orgId: 1
receiver: default-channel-name
group_by: ['...']
routes:
- receiver: grafana-alerts
object_matchers:
- ['app', '=', 'demo-app']
You can keep all resources in one ConfigMap or give each alerting resource its own.
A ConfigMap containing all the defined resources should look as follows:
kind: ConfigMap
metadata:
name: grafana-alerting
namespace: monitoring
labels:
grafana_alert: "1"
data:
grafana-alerting.yaml: |-
apiVersion: 1
templates:
- orgID: 1
name: slack.message
template: |-
# add here your notification template
policies:
- orgId: 1
receiver: default-channel-name
group_by: ['...']
routes:
- receiver: grafana-alerts
object_matchers:
- ['app', '=', 'demo-app]
contactPoints:
- orgId: 1
name: grafana-alerts
receivers:
- uid: grafana
type: slack
settings:
username: grafana_bot
url: incoming_webhook_url
title: |
{{ template "slack.title" . }}
text: |
{{ template "slack.message" . }}
groups:
- orgId: 1
name: demo-app
folder: Log filtering
interval: 5m
rules:
- title: Exception in demo-app
# add here exported alert rules
Apply these manifests, then open the Grafana Alerting page to check that every resource has been provisioned properly. Each of them should carry the Provisioned label.
Note: You cannot edit provisioned resources from the UI – they are immutable.
After that, go to the contact points page and run a test of the contact point:
If a test notification lands in your Slack channel, well done – everything is configured correctly, and you can now start receiving alerts.
Below is a preview of the message delivered to the Slack channel. Since it is a test notification, it is a trimmed-down version that lacks a log message and links to the dashboard or panel.
Summary
One useful lesson I learned along the way: when you begin configuring Grafana Alerting, build all the key resources in the Grafana UI first, and then save them with the Export button. As of the 10.1 release, the UI lets you export not just alert rules, but contact points and notification policies as well. When I was starting out with Grafana Alerting, these export options did not exist, and I spent a lot of additional time debugging YAML files to track down mistakes. Following this advice can spare you time and frustration during setup.
That said, provisioning through Kubernetes is an excellent choice, as it lets you describe your monitoring and observability setup directly in your GitOps repository. Grafana Alerting gives you a flexible, highly customizable way to configure alerts, which is essential for responding fast when they fire. This guide should help you get a better grasp of the Grafana Alerting concepts and set up basic alerts for a Slack channel.