Ending the hour-long monitoring blind spot after every Prometheus restart

Overview

A payments processor could not restart Prometheus during business hours because replay took long enough to leave a substantial monitoring gap. That made routine patching a scheduling problem. AceMQ provides ongoing support covering restart behavior, retention, and availability of the metrics path itself.

Challenge

Each Prometheus instance carried a very high active series count, so WAL replay on startup ran long and the instance served no queries until it finished. Any node eviction, config reload requiring restart, or version upgrade produced a blind spot that the risk team would not accept during trading hours. The obvious workaround — running a second replica — had been attempted but produced inconsistent query results because the replicas were not deduplicated.

Environment

Prometheus pairs per region on Kubernetes with long-term storage, high active series counts, regulated change windows.

Approach

We addressed the blind spot from two directions: reducing the replay work itself and making the restart invisible to queriers. Cutting series count through relabeling shortened replay directly. Adding proper replica deduplication in front of the pair meant one instance could restart while the other served, which turned restarts into ordinary operations.

Solution

• 24/7 support with a 15-minute emergency SLA and senior engineers who hold context on each region's topology
• Series count reduced through relabel-based dropping of unused metrics and consolidation of duplicate exporters
• Highly available Prometheus pairs fronted by a deduplicating query layer so one instance can restart transparently
• Retention and block duration tuned to reduce WAL size and shorten replay without losing query range
• Restart, upgrade, and config reload procedures documented and rehearsed against the regulated change window
• Recurring review of series growth so replay time stays bounded as new services onboard

Outcome

Technologies