Alert2Dash is a Kubernetes Operator that automatically generates Grafana dashboards from Prometheus alerting rules. It simplifies the process of visualizing and monitoring your alerts by creating customized dashboards that stay synchronized with your alerting definitions.
- Automatic Dashboard Generation: Convert PrometheusRules directly into meaningful Grafana dashboards
- Zero-Drift Guarantee: Dashboards automatically update when alert definitions change
- Template Support: Use custom Jsonnet templates for advanced dashboard customization: panel sizes, and grouping etc.
- Kubernetes-Native: Fully integrated with Kubernetes resource model and lifecycle management
Note: This operator is designed to work alongside:
- prometheus-operator - for managing Prometheus rules and alerts
- grafana-operator - for managing Grafana dashboards
Alert2Dash Operator follows a Kubernetes-native approach:
- It monitors PrometheusRule resources in your cluster
- When an AlertDashboard custom resource is created, it uses label selectors to identify relevant PrometheusRules
- It extracts metrics from alert expressions and analyzes their query patterns
- It generates dashboard JSON using predefined template
- The generated JSON is stored in ConfigMaps with proper labels
- The Grafana Operator detects these ConfigMaps and creates actual dashboards in Grafana
Here's the diagram with Grafana Dashboards as output boxes from Grafana Operator:
┌─────────────────┐
│ PrometheusRule │
│ labels: │──┐ ┌───────────────────┐ ┌────────────────┐ ┌─────────────┐ ┌────────────────┐
│ team: foo │ │ │ AlertDashboard-1 │ │ ConfigMap-1 │ │ Grafana │ │ Dashboard │
└─────────────────┘ ├────▶ │ selector: │──▶ │ (dash.json) │──────▶│ Operator │─────▶│ "Team Foo" │
┌─────────────────┐ │ │ team: foo │ └────────────────┘ │ │ └────────────────┘
│ PrometheusRule │──┘ └───────────────────┘ │ │
│ labels: │ │ │
│ team: foo │ │ │ ┌────────────────┐
└─────────────────┘ │ │ │ Dashboard │
│ │────▶ │ "Team Bar" │
┌─────────────────┐ │ │ └────────────────┘
│ PrometheusRule │──┐ ┌───────────────────┐ ┌────────────────┐ │ │
│ labels: │ │ │ AlertDashboard-2 │ │ ConfigMap-2 │ │ │
│ team: bar │ ├────▶ │ selector: │──▶ │ (dash.json) │──────▶│ │
└─────────────────┘ │ │ team: bar │ └────────────────┘ │ │
┌─────────────────┐ │ └───────────────────┘ │ │
│ PrometheusRule │──┘ └─────────────┘
│ labels: │
│ team: bar │
└─────────────────┘
Here's an example of an AlertDashboard custom resource that creates a dashboard for a specific team:
apiVersion: monitoring.krutsko.com/v1alpha1
kind: AlertDashboard
metadata:
name: team-foo
namespace: monitoring
spec:
# Selector for the team's alerts
metadataLabelSelector:
matchLabels:
team: foo
# Selector for the alert rules to include
ruleLabelSelector:
matchLabels:
severity: critical
dashboardConfig:
configMapNamePrefix: "grafana-dashboard"This example:
- Creates a dashboard named "team-foo" in the monitoring namespace
- Selects alerts specifically for Team Foo using the
teamlabel - Includes only critical severity alerts
- Creates ConfigMaps with the prefix "grafana-dashboard"
You can also use matchExpressions for more advanced label selection with operators like In, NotIn, Exists, and DoesNotExist:
apiVersion: monitoring.krutsko.com/v1alpha1
kind: AlertDashboard
metadata:
name: multi-team-dashboard
namespace: monitoring
spec:
# Select PrometheusRules from multiple teams
metadataLabelSelector:
matchExpressions:
- key: team
operator: In
values: ["sre", "platform"]
# Select alerts with severity of either critical or warning
ruleLabelSelector:
matchExpressions:
- key: severity
operator: In
values: ["critical", "warning"]
dashboardConfig:
configMapNamePrefix: "grafana-dashboard"The operator will automatically:
- Find all PrometheusRules with matching labels
- Extract metrics from alert expressions
- Generate a Grafana dashboard
- Create a ConfigMap with the dashboard JSON
- The Grafana Operator will then create the actual dashboard in Grafana
You can exclude specific alert rules from being included in the dashboard by adding the alert2dash_exclude_rule: "true" label to individual alert rules or PrometheusRule resources:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: my-alerts
spec:
groups:
- name: example
rules:
- alert: IncludedAlert
expr: up == 0
- alert: ExcludedAlert
expr: down == 0
labels:
alert2dash_exclude_rule: "true" # This alert will not appear in dashboardsThe operator matches labels at multiple levels:
- PrometheusRule metadata labels - labels on the PrometheusRule resource
- Rule group labels - labels defined in the rule group
- Individual alert labels - labels on specific alert rules
When checking ruleLabelSelector, the operator merges labels from all levels. For example, if a PrometheusRule has a team: platform label at the group level, all alerts in that group will match a selector for team: platform.
The configMapNamePrefix field is required and must be a valid Kubernetes DNS name.
Option 1: Helm Chart
Deploy the Alert2Dash Operator easily in your cluster using Helm:
helm upgrade -i alert2dash oci://ghcr.io/krutsko/helm-charts/alert2dash-operator -n alert2dash-systemOption 2: Using the installer
Users can just run kubectl to install the project, i.e.:
- Latest release
kubectl apply -f https://github.com/krutsko/alert2dash-operator/releases/download/latest/install.yaml- Specific version
kubectl apply -f https://github.com/krutsko/alert2dash-operator/releases/download/vX.Y.Z/install.yaml- go version v1.22.0+
- docker version 17.03+.
- kubectl version v1.11.3+.
- Access to a Kubernetes v1.11.3+ cluster.
- Install Instances of Custom Resources:
CRDs:
kubectl apply -f config/crd/basesAll:
kubectl apply -f config/samples/AlertDashboard:
kubectl apply -f config/samples/monitoring_v1alpha1_alertdashboard.yamlBuild and push your image to the location specified by IMG:
make docker-build docker-push IMG=ghcr.io/krutsko/alert2dash-operator:tagBuild image without pushing it to the registry:
make docker-build IMG=ghcr.io/krutsko/alert2dash-operator:tagNOTE: This image ought to be published in the personal registry you specified. And it is required to have access to pull the image from the working environment. Make sure you have the proper permission to the registry if the above commands don't work.
Install the CRDs into the cluster:
make installDeploy the Manager to the cluster with the image specified by IMG:
make deploy IMG=ghcr.io/krutsko/alert2dash-operator:tagLoad docker image to load kind cluster:
kind load docker-image ghcr.io/krutsko/alert2dash-operator:tag -n <cluster>Load docker image to load kind cluster:
kind load docker-image <some-registry>/alert2dash-operator:tag -n v1.23NOTE: If you encounter RBAC errors, you may need to grant yourself cluster-admin privileges or be logged in as admin.
Create instances of your solution You can apply the samples (examples) from the config/sample:
kubectl apply -k config/samples/NOTE: Ensure that the samples has default values to test it out.
helm pull oci://ghcr.io/krutsko/helm-charts/alert2dash-operatorDelete the instances (CRs) from the cluster:
kubectl delete -k config/samples/Delete the APIs(CRDs) from the cluster:
make uninstallUnDeploy the controller from the cluster:
make undeployFollowing are the steps to build the installer and distribute this project to users.
- Build the installer for the image built and published in the registry:
make build-installer IMG=ghcr.io/krutsko/alert2dash-operator:tagNOTE: The makefile target mentioned above generates an 'install.yaml' file in the dist directory. This file contains all the resources built with Kustomize, which are necessary to install this project without its dependencies.
- Using the installer
Users can just run kubectl apply -f to install the project, i.e.:
kubectl apply -f https://raw.githubusercontent.com/<org>/alert2dash-operator/<tag or branch>/dist/install.yamlContributions are welcome! Please feel free to submit a Pull Request.
NOTE: Run make help for more information on all potential make targets
More information can be found via the Kubebuilder Documentation