An honest map of what your APM can actually see
KubernetesOpenTelemetryRoughly a fifth of business-critical code paths turned out to be invisible to APM despite agents being present. With custom instrumentation added to the highest-priority gaps, the class of outage that…
Overview
A payments processor believed their estate was fully instrumented until an outage in an uncovered component took hours to locate. AceMQ assessed real coverage across services, dependencies, and background work.
Challenge
Coverage looked complete because every deployed service had an agent attached. What the inventory missed was that several critical code paths were invisible anyway: background workers and scheduled jobs produced no transactions because nothing marked their entry points, custom database access layers were not recognized by the auto-instrumentation, and outbound calls through an internal HTTP client wrapper appeared as generic external segments with no target attribution. The map showed coverage; the reality had holes exactly where debugging was hardest.
Environment
Hybrid estate with Java and Node.js services, scheduled batch work, custom data access and HTTP client libraries.
Approach
Rather than counting agents, the assessment checks what telemetry each service actually produces against what it actually does. We compare the deployment inventory to reporting entities, inspect transaction and segment data per service for the expected shapes, and specifically probe the areas auto-instrumentation is known to miss.
Solution
- 1Deployment inventory reconciled against reporting entities to find services that are deployed but silently not reporting
- 2Per-service transaction and segment data inspected for expected shapes rather than assuming an attached agent means coverage
- 3Background workers, scheduled jobs, and consumer loops checked specifically, since these rarely instrument automatically
- 4Custom data access and HTTP client wrappers identified as instrumentation blind spots with explicit remediation defined
- 5Alert and dashboard coverage mapped against business-critical paths to find monitored components with no alerting
- 6Prioritized remediation backlog ordered by blast radius, with effort estimated per item
Outcome
Roughly a fifth of business-critical code paths turned out to be invisible to APM despite agents being present. With custom instrumentation added to the highest-priority gaps, the class of outage that previously took hours to locate is now visible in the first minutes.
Technologies
KubernetesOpenTelemetryRelated Use Cases
New Relic Telemetry Volume and Cost Governance
Consulting engagement to govern ingested telemetry volume and user allocation so observability spend reflects operational value.
New Relic Distributed Tracing Support
Ongoing support for distributed tracing across a mixed New Relic and OpenTelemetry estate where traces broke at instrumentation boundaries.
Ready for a New Relic Health Check?
AceMQ's senior New Relic engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.