Overview
A typical engagement starts with a platform team that built its monitoring on a single Prometheus and a single Grafana in its Kubernetes cluster. The stack works, but it has no replica, the alerting path has never been tested end to end, and the engineer who built it is also the one paged when it fails.
Challenge
Prometheus local storage is a single node and is not clustered or replicated, so the team's metrics stop when that pod does. Grafana runs on a default database file, alert rules and dashboards were created in the UI, and nobody can say what a rebuilt instance would contain. Security policy keeps the telemetry and the data source credentials inside the cluster, which rules out most hosted options.
Environment
One or more Kubernetes clusters, managed or self-run, with the Prometheus Operator or a kube-prometheus style deployment, an Alertmanager, Grafana with a few dozen dashboards and a paging tool. Object storage is available for long-term retention.
Approach
AceMQ engineers, with partner specialists, review the deployment and agree the Tier 3 scope in writing. They add a second Prometheus server and an Alertmanager cluster, move Grafana to a shared database with more than one instance, and bring dashboards, alert rules and contact points into the team's repository. Access to the cluster follows the customer's change process and is logged. Then AceMQ takes the pager.
Solution
- Replicated Prometheus servers with external labels, and Alertmanager run as a cluster with Prometheus pointed at every instance
- Retention set by time and by size, with remote write to the long-term storage layer the team chooses
- Grafana instances behind a load balancer on a shared database, with alerting high availability configured and tested
- Dashboards, folders, data sources, alert rules and contact points provisioned from the customer's repository, with a written policy on UI edits
- Monitoring of the monitoring: active series, rule evaluation, target health and the alert delivery path, with a test notification after every change
- Upgrades planned against the Prometheus long-term support lines and Grafana's nine-month support window, rehearsed on a copy
Outcome
The engagement delivers a replicated, monitored stack operated by AceMQ inside the customer's cluster, a repository that describes it, tested alerting paths, written upgrade plans and runbooks that stay with the customer. The customer's engineers keep building dashboards and rules, and are no longer the first call when the monitoring itself fails.
Technologies
Grafana
PrometheusAlertmanager
Kubernetes
Amazon S3
PostgreSQL
