When the monitoring agent becomes the performance problem
KubernetesLinuxThe memory growth stopped and p99 latency returned to its pre-upgrade level. Transaction data became usable again because it groups by meaningful endpoint rather than by every distinct URL.
Overview
A commerce platform saw p99 latency climb and container memory grow steadily after an agent upgrade. AceMQ remediated the agent configuration and JVM interaction — the customer's side of the boundary, which vendor support does not tune.
Challenge
Transaction naming was falling back to a per-URL scheme because the framework instrumentation did not recognize a custom router, so every distinct path produced a separate transaction. That drove metric grouping issues and a large in-agent name cache that grew with traffic. Separately, distributed tracing had been enabled at full sampling on a very high-throughput service, adding measurable per-request overhead and steady allocation pressure that showed up as longer garbage collection pauses.
Environment
Java services on Kubernetes with a custom routing framework, high request throughput, memory-limited containers.
Approach
We separated agent-attributable overhead from application behavior by running controlled comparisons with the agent enabled and disabled under identical load. Once agent cost was quantified, we corrected transaction naming so the agent stopped generating unbounded names, and set sampling to a level appropriate for the service's throughput.
Solution
- 1Controlled load comparison with agent enabled and disabled to quantify actual overhead instead of estimating it
- 2Transaction naming corrected through explicit API calls so the custom router produces bounded, meaningful names
- 3Metric grouping issues resolved, eliminating the unbounded in-agent name cache driving memory growth
- 4Distributed tracing sampling reduced on the highest-throughput service while retained fully on error paths
- 5JVM heap and container memory limits realigned to account for real agent footprint rather than pre-agent sizing
- 6Agent version upgrade path tested in staging under production-like load before rollout
Outcome
The memory growth stopped and p99 latency returned to its pre-upgrade level. Transaction data became usable again because it groups by meaningful endpoint rather than by every distinct URL.
Technologies
KubernetesLinuxRelated Use Cases
New Relic Distributed Tracing Support
Ongoing support for distributed tracing across a mixed New Relic and OpenTelemetry estate where traces broke at instrumentation boundaries.
New Relic Telemetry Volume and Cost Governance
Consulting engagement to govern ingested telemetry volume and user allocation so observability spend reflects operational value.
Facing a New Relic Production Issue?
AceMQ's senior New Relic engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.