Overview
A financial services group had metrics in one tool, logs in another, and traces in a third, with no shared identifiers between them. AceMQ designed the consolidation onto a Grafana-fronted stack where an engineer can move from a metric spike to the relevant logs and traces without re-deriving context.
Challenge
The three signal types used different names for the same things — service names differed by casing and suffix, environments were labeled inconsistently, and there was no shared trace or request identifier in the logs. Correlating an alert with its logs meant a human translating between naming schemes under time pressure. Duplicated agents across the estate also meant the same host was being scraped several times over.
Environment
Hybrid estate with Kubernetes workloads and legacy virtual machines, multiple regulated environments requiring data residency separation.
Approach
The design work started with a label taxonomy rather than tooling, because correlation is a naming problem before it is a product problem. We defined a required label set applied uniformly across metrics, logs, and traces, specified trace context propagation so log lines carry the trace identifier, and then designed the collection topology that enforces it at the agent layer.
Solution
- Mandatory label taxonomy defined for service, environment, region, and team, applied identically across metrics, logs, and traces
- Trace context propagation designed so log lines carry trace and span identifiers, making metric-to-log-to-trace pivots one click
- Collection topology rationalized to a single agent per host, removing duplicate scrapes and their storage cost
- Data residency boundaries enforced at the collection layer so regulated regions never ship telemetry across borders
- Retention tiers defined per signal type, keeping high-resolution data short and downsampled data long
- Migration sequenced service by service with dual-emit periods so no team loses visibility mid-transition
Outcome
Engineers now pivot from an alert to the correlated logs and trace directly in Grafana instead of manually translating identifiers. Removing duplicate collection cut telemetry volume noticeably, and the shared taxonomy has held as new services onboard.
Technologies
Grafana
PrometheusLokiTempo
Kubernetes
Amazon S3
Terraform
PostgreSQL
Linux
