Working out why the global view disagrees with every site view

Overview

A manufacturer running Prometheus at each plant plus a global federation layer found the global dashboards disagreed with local ones. AceMQ assessed the topology to determine whether federation was the right pattern at their scale.

Challenge

Hierarchical federation was being used to pull raw series rather than aggregates, so the global instance was scraping an enormous payload from each site on every interval. Scrapes regularly exceeded the timeout and returned partial data, which is why global and local views disagreed. Sites with intermittent WAN links produced gaps that looked like plant outages. Nobody could tell which numbers to trust during an actual production incident.

Environment

Prometheus at multiple manufacturing sites with intermittent WAN connectivity to a central observability environment.

Approach

The assessment measured what federation was actually transferring against the scrape timeout budget, which made the partial-scrape problem plain. We then evaluated whether the customer's requirements were better served by remote write with a central store than by federation, and modeled the bandwidth and storage implications of each option against their real series counts.

Solution

• Federation payload size and duration measured per site and compared against the configured scrape timeout
• Partial-scrape and gap incidents correlated with WAN link events to separate network problems from configuration ones
• Recording rules designed at each site so only aggregates leave the plant, cutting federation payload by orders of magnitude
• Remote write with a central long-term store evaluated as an alternative, with bandwidth and storage modeled on real series counts
• Buffering and retry behavior specified so a WAN outage produces backfilled data rather than a permanent hole
• Per-site retention defined so local engineers keep full resolution even when only aggregates go central

Outcome

Technologies