Named senior engineers on call when the heap turns red

Overview

A multi-tenant analytics vendor kept losing nodes to long stop-the-world garbage collection pauses during customer-facing dashboard refreshes. AceMQ provides continuous support with a 15-minute emergency SLA and named senior engineers who already know the cluster topology.

Challenge

Parent circuit breaker exceptions surfaced as failed searches for whichever tenant happened to be querying at the time, which made the problem look random. Deep terms aggregations over high-cardinality fields and unbounded date-histogram buckets were pulling large amounts of field data into heap, and a handful of oversized queries could push a node past the breaker and trigger a GC pause long enough for the master to drop it from the cluster.

Environment

Elasticsearch on Kubernetes across three availability zones, mixed tenant query load, dashboards refreshing on short intervals.

Approach

Support engagements start with hot threads and GC log analysis during a live incident, not a post-mortem summary. We correlate breaker trips with the slow log to identify the specific query shapes responsible, then work with the customer's engineers to bound them at the API layer rather than simply raising breaker limits.

Solution

• 24/7 incident response with a 15-minute emergency SLA and no tier-1 triage layer between the customer and a senior engineer
• Hot threads, GC log, and node stats analysis captured during live breaker events rather than reconstructed afterward
• Search slow log tuning to surface the aggregation shapes that actually drive heap growth
• Guidance on bounding terms aggregation size, adding search.max_buckets limits, and using composite aggregations for deep pagination
• Field data and query cache sizing reviewed against real workload rather than defaults
• Rolling restart and node replacement procedures validated with the customer during maintenance windows

Outcome

Technologies