Stop one bad query from taking the whole Trino cluster down
Worker crashes stop, and queries that exceed their allowance fail with a clear resource error rather than taking peers down with them. Scheduled pipelines finish predictably because ad-hoc load can no…
Overview
A single unconstrained join in Trino can exhaust worker heap and take down nodes that dozens of other queries depend on. AceMQ remediates the immediate instability and then puts the resource controls in place that keep one analyst's query from becoming everyone's outage.
Challenge
Trino's failure mode under memory pressure is abrupt. A cross join with a bad predicate, a broadcast join where the build side turned out to be far larger than the statistics suggested, or an ORDER BY over an unfiltered fact table will push worker heap past its limit and kill the JVM. Without resource groups, there is nothing between one query and the whole cluster, and retries make the next attempt worse.
Environment
Starburst Enterprise on Kubernetes or VMs, serving mixed ad-hoc and scheduled analytical workloads.
Approach
AceMQ starts from the JVM and query logs to identify the specific operators and queries responsible, then applies per-query and per-node memory limits so the offending query fails cleanly instead of killing the worker. Spill-to-disk is configured for the joins and sorts that legitimately exceed memory, and resource groups partition capacity so scheduled work is insulated from ad-hoc exploration.
Solution
- 1JVM heap, GC log, and query event analysis to attribute crashes to specific operators and query shapes
- 2query.max-memory, query.max-memory-per-node, and memory.heap-headroom tuning matched to real concurrency
- 3Spill-to-disk configuration with appropriate local storage for joins, aggregations, and sorts that legitimately exceed memory
- 4Resource group definitions separating scheduled pipelines, BI tools, and ad-hoc analyst queries with hard concurrency caps
- 5Join distribution and reordering review, including broadcast threshold correction where statistics are misleading the optimizer
- 6Coordinator sizing and worker node profile review, with graceful shutdown configured for autoscaling clusters
Outcome
Worker crashes stop, and queries that exceed their allowance fail with a clear resource error rather than taking peers down with them. Scheduled pipelines finish predictably because ad-hoc load can no longer consume the whole cluster.
Technologies
Related Use Cases
Starburst Federated Query Pushdown Tuning
Making predicates, aggregates, and joins execute at the source connector instead of pulling full tables into Trino workers.
Starburst Query Latency Support
Named-engineer support for Starburst and Trino latency regressions, connector failures, and concurrency problems in production.
Facing a Starburst Production Issue?
AceMQ's senior Starburst engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.