Push the work down to the source instead of dragging the source into Trino

Overview

Starburst's value depends on connectors doing work at the source. When pushdown silently fails, Trino compensates by streaming entire tables across the network and filtering them in worker memory, which is the slowest possible way to answer the query. AceMQ finds where pushdown is breaking and fixes the conditions that break it.

Challenge

Pushdown failures are quiet. A query that should send a filtered, aggregated request to Snowflake or PostgreSQL instead issues a full table scan because a CAST on the predicate column defeated the connector's rewrite, or a function has no source equivalent, or the join key types differ across catalogs. Teams see slow queries and add workers, which increases cost without touching the actual problem.

Environment

Starburst Enterprise or Galaxy federating relational sources, object storage, and warehouse connectors across on-premises and cloud.

Approach

AceMQ reads EXPLAIN and EXPLAIN ANALYZE output for the slowest recurring queries to see exactly which operators the connector accepted and which stayed in Trino. Each blocked pushdown is traced to its cause — type mismatch, unsupported function, missing statistics — and fixed at the query, view, or connector configuration level. Cost-based optimization is then given the statistics it needs to pick sane join orders.

Solution

• EXPLAIN ANALYZE review of the slowest recurring queries to identify which predicates, aggregates, and joins failed to push down
• Type alignment across catalogs so join keys and filter columns stop forcing casts that defeat connector rewrites
• Connector-level configuration for predicate, aggregate, join, and TopN pushdown per source system
• Table statistics collection and ANALYZE scheduling so the cost-based optimizer picks the right join order and distribution
• Dynamic filtering configuration for star-schema joins across federated sources
• Materialized view and cache service design for the small number of cross-source joins that genuinely cannot push down

Outcome

Technologies