When the monitoring agent becomes the performance problem
KubernetesLinuxThe memory growth stopped and p99 latency returned to its pre-upgrade level. Transaction data became usable again because it groups by meaningful endpoint rather than by every distinct URL.
Overview
A commerce platform saw p99 latency climb and container memory grow steadily after an agent upgrade. AceMQ remediated the agent configuration and JVM interaction — the customer's side of the boundary, which vendor support does not tune.
Challenge
Transaction naming was falling back to a per-URL scheme because the framework instrumentation did not recognize a custom router, so every distinct path produced a separate transaction. That drove metric grouping issues and a large in-agent name cache that grew with traffic. Separately, distributed tracing had been enabled at full sampling on a very high-throughput service, adding measurable per-request overhead and steady allocation pressure that showed up as longer garbage collection pauses.
Environment
Java services on Kubernetes with a custom routing framework, high request throughput, memory-limited containers.
Approach
We separated agent-attributable overhead from application behavior by running controlled comparisons with the agent enabled and disabled under identical load. Once agent cost was quantified, we corrected transaction naming so the agent stopped generating unbounded names, and set sampling to a level appropriate for the service's throughput.
Solution
- 1Controlled load comparison with agent enabled and disabled to quantify actual overhead instead of estimating it
- 2Transaction naming corrected through explicit API calls so the custom router produces bounded, meaningful names
- 3Metric grouping issues resolved, eliminating the unbounded in-agent name cache driving memory growth
- 4Distributed tracing sampling reduced on the highest-throughput service while retained fully on error paths
- 5JVM heap and container memory limits realigned to account for real agent footprint rather than pre-agent sizing
- 6Agent version upgrade path tested in staging under production-like load before rollout
Outcome
The memory growth stopped and p99 latency returned to its pre-upgrade level. Transaction data became usable again because it groups by meaningful endpoint rather than by every distinct URL.
Technologies
KubernetesLinuxRelated Use Cases
New Relic Distributed Tracing Support
Ongoing support for distributed tracing across a mixed New Relic and OpenTelemetry estate where traces broke at instrumentation boundaries.
New Relic Telemetry Volume and Cost Governance
Consulting engagement to govern ingested telemetry volume and user allocation so observability spend reflects operational value.
Elasticsearch Heap Pressure and Circuit Breaker Support
Ongoing 24/7 support for an Elasticsearch estate suffering repeated parent circuit breaker trips and long garbage collection pauses under aggregation load.
Memcached Slab Calcification and Eviction Remediation
Emergency remediation of a Memcached tier evicting hot keys while reporting free memory, traced to slab class allocation.
New Relic Instrumentation Coverage Assessment
Assessment of which services, dependencies, and code paths are genuinely covered by APM instrumentation versus assumed to be.
Redis Latency Spike Remediation from Fork Stalls
Eliminating periodic multi-hundred-millisecond latency spikes traced to RDB snapshot fork stalls amplified by transparent huge pages.
Facing a New Relic Production Issue?
AceMQ's senior New Relic engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.