Stop one bad query from taking the whole Trino cluster down

Overview

A single unconstrained join in Trino can exhaust worker heap and take down nodes that dozens of other queries depend on. AceMQ remediates the immediate instability and then puts the resource controls in place that keep one analyst's query from becoming everyone's outage.

Challenge

Trino's failure mode under memory pressure is abrupt. A cross join with a bad predicate, a broadcast join where the build side turned out to be far larger than the statistics suggested, or an ORDER BY over an unfiltered fact table will push worker heap past its limit and kill the JVM. Without resource groups, there is nothing between one query and the whole cluster, and retries make the next attempt worse.

Environment

Starburst Enterprise on Kubernetes or VMs, serving mixed ad-hoc and scheduled analytical workloads.

Approach

AceMQ starts from the JVM and query logs to identify the specific operators and queries responsible, then applies per-query and per-node memory limits so the offending query fails cleanly instead of killing the worker. Spill-to-disk is configured for the joins and sorts that legitimately exceed memory, and resource groups partition capacity so scheduled work is insulated from ad-hoc exploration.

Solution

• JVM heap, GC log, and query event analysis to attribute crashes to specific operators and query shapes
• query.max-memory, query.max-memory-per-node, and memory.heap-headroom tuning matched to real concurrency
• Spill-to-disk configuration with appropriate local storage for joins, aggregations, and sorts that legitimately exceed memory
• Resource group definitions separating scheduled pipelines, BI tools, and ad-hoc analyst queries with hard concurrency caps
• Join distribution and reordering review, including broadcast threshold correction where statistics are misleading the optimizer
• Coordinator sizing and worker node profile review, with graceful shutdown configured for autoscaling clusters

Outcome

Technologies