Fixing the monitoring stack that fails at exactly the wrong moment

Overview

A telecom operator's Grafana instance would slow to a crawl during major incidents, which is precisely when the network operations team depended on it. AceMQ remediated the failure mode rather than adding more replicas.

Challenge

The pattern was self-inflicted: during an incident, dozens of engineers opened the same set of heavy dashboards, each firing many concurrent range queries at the datasource. The backend could not keep up, panels timed out, engineers hit refresh, and the retry storm made it worse. Grafana's own session and annotation writes to its backing database added contention on top. Nothing was broken in isolation; the system only failed when it was needed.

Environment

Grafana on Kubernetes backed by a relational database, querying Prometheus and long-term metric storage across several regions.

Approach

We reproduced the failure by load-testing the incident access pattern instead of the steady-state one. That made the specific expensive panels obvious. We then reduced per-dashboard query cost, added caching in front of the datasource, and separated the Grafana backing database load from the query path so the UI stays responsive even when datasources are slow.

Solution

• Load test built around the real incident access pattern — many concurrent viewers on the same dashboards — rather than average traffic
• Expensive panels identified and rewritten using recording rules so the heavy aggregation happens once at ingest, not per viewer
• Query result caching added in front of the datasource with TTLs tuned per panel refresh interval
• Dashboard refresh intervals and default time ranges revised so an idle open tab stops generating continuous load
• Grafana backing database tuned and connection pooling corrected to remove contention from annotation and session writes
• Panel-level query timeouts set so a slow datasource degrades one panel instead of hanging the whole dashboard

Outcome

Technologies