Getting the on-call rotation back to alerts that mean something

Overview

A SaaS provider's on-call engineers were acknowledging alerts reflexively because most of them required no action. AceMQ provides ongoing support to keep the monitor estate meaningful, working alongside the customer's team on their own Datadog organization.

Challenge

Monitors had accumulated over years with no retirement process. Many used static thresholds set against traffic levels that no longer existed, so they fired on every normal peak. Composite monitors created alert storms where one underlying failure produced dozens of pages. Multi-alert monitors grouped by a tag that had since become high cardinality, generating one alert per instance rather than one per service. Nobody owned the estate, so nothing was ever removed.

Environment

Datadog monitoring a multi-region AWS and Kubernetes estate, alerts routed into an on-call rotation.

Approach

We work from alert history rather than opinion: which monitors fired, how often, and whether anyone did anything as a result. Monitors that never lead to action get retired or downgraded to a dashboard. The ones that matter get thresholds that reflect current behavior, appropriate grouping, and downtime rules that suppress the predictable noise.

Solution

• 24/7 support with a 15-minute emergency SLA and named senior engineers embedded in the customer's alert review cadence
• Alert history analyzed by monitor to separate the ones that drive action from the ones that are always acknowledged and closed
• Static thresholds replaced with anomaly or forecast-based conditions where the underlying signal has real seasonality
• Multi-alert grouping corrected so alerts arrive per service rather than per ephemeral instance
• Composite monitor logic restructured so one root failure produces one page instead of a storm
• Monitors defined as code with owners attached, so unowned monitors are visible and removable

Outcome

Technologies