Back to all use cases
Energy / UtilitiesSupportHybrid

Alerts that fire when they should and stay quiet when they shouldn't

EU
Energy Utility Operator

Overview

An energy utility relies on Grafana alerting for operational monitoring across generation and distribution sites. AceMQ provides continuous support covering alert rule behavior, notification routing, and datasource stability.

Challenge

After migrating from legacy alerting to unified alerting, the utility had rules that fired inconsistently. Some alerts never resolved because the underlying series stopped being reported and the rule had no no-data handling. Others flapped because the evaluation interval was shorter than the scrape interval, so the rule regularly evaluated against a gap. Notification policies had grown into a tree nobody fully understood, and a mislabeled route meant a class of alerts had been silently going nowhere.

Environment

Grafana in a hybrid deployment monitoring on-premises SCADA-adjacent infrastructure and cloud services, alerting into on-call rotation tooling.

Approach

Support treats alert rules as production code. We review no-data and error handling on every rule, align evaluation intervals with the underlying scrape and recording rule cadence, and validate that every notification policy path actually reaches a human. Rule changes are tested against historical data before they go live.

Solution

  • 24/7 support with a 15-minute emergency SLA and named senior engineers who know the alert estate
  • Every alert rule audited for no-data and execution-error handling so silent failures become visible
  • Evaluation intervals and for-durations aligned to scrape and recording rule cadence to eliminate gap-driven flapping
  • Notification policy tree simplified and every route verified end to end against a test alert
  • Alert rules and contact points managed as provisioned configuration so changes are reviewable and reversible
  • Recurring review of alert volume and acknowledgement rates to retire rules nobody acts on

Outcome

Alert noise dropped substantially once flapping rules were fixed, and the routes that had been going nowhere now reach on-call. Engineers trust the alerts again, which shows up as faster acknowledgement times.

Technologies

GrafanaPrometheusAlertmanagerLinux

Ready to Get Started?

Whether you need architecture advisory, 24/7 support, or full managed services, AceMQ has the expertise to help.

Contact Us