Stopping a cardinality explosion before it takes monitoring down for good

Overview

A commerce platform lost metrics visibility when their Prometheus servers began getting OOM-killed within minutes of starting. The trigger was a routine application deployment; the cause was a single new label.

Challenge

A team had added a user identifier as a label on a request counter. Every distinct user created a new time series, so active series count grew without bound during traffic hours. Prometheus consumed memory until the kernel killed it, and on restart it had to replay a write-ahead log that had grown enormous, which took long enough that the process was killed again before it finished. The monitoring system was in a crash loop it could not recover from on its own, and the team had no metrics at all while it was down.

Environment

Prometheus on Kubernetes scraping several hundred targets, feeding Grafana dashboards and Alertmanager routes.

Approach

First priority was getting a Prometheus process to stay up, which meant breaking the WAL replay loop before addressing the metric itself. We brought the server up in a state where it could complete replay, identified the offending series by cardinality, dropped it at the relabel stage so it stopped being ingested, and then put guardrails in place so the same class of mistake cannot silently repeat.

Solution

• Broke the crash loop by giving the process enough headroom to complete WAL replay, then confirmed the block was memory not corruption
• Ranked series by cardinality using TSDB status to identify the offending label within minutes rather than by inspection
• Dropped the unbounded label at metric_relabel_configs so it stopped being ingested regardless of what the application emits
• Applied sample_limit and label_limit per scrape config so a single bad target cannot take down the server again
• Added a cardinality alert on active series growth rate that fires before memory becomes critical
• Worked with the application team on the correct pattern — user identity belongs in logs and traces, not metric labels

Outcome

Technologies