Custom metric cardinality governance that survives the next deployment

Overview

A commerce platform's observability bill was growing faster than its traffic, driven mostly by custom metrics nobody had reviewed. AceMQ helped them understand where the volume came from and put controls in place that hold as new services ship.

Challenge

Custom metric billing is driven by distinct tag combinations, and several teams had added deployment identifiers, pod names, and customer identifiers as tags. A single metric with a pod-name tag on a high-churn deployment generated an enormous number of billable series that no dashboard ever queried. Log ingest had the same problem in a different shape: debug-level logs from a chatty framework accounted for a large share of indexed volume and were never searched.

Environment

Datadog across a multi-account AWS estate with Kubernetes workloads, high deployment frequency, multiple product teams.

Approach

We do not operate the vendor's platform; we help customers control what they send to it. The engagement quantified volume by origin, established which telemetry was actually queried, and then implemented controls at the emission and collection layers so reductions do not quietly reverse on the next deploy.

Solution

• Custom metric volume attributed to specific metrics and tag keys, ranking by billable series contribution
• Query usage cross-referenced so metrics and tags nobody has ever queried are identified as safe to drop
• Tag governance defined — identifiers with unbounded values belong in logs or traces, never in metric tags
• Metric and tag filtering applied at the collection layer so the rule is enforced regardless of what applications emit
• Log indexing policy split so high-volume, low-value logs are retained without being indexed for search
• Trace sampling strategy revised to keep error and slow traces at full fidelity while sampling routine successful requests

Outcome

Technologies