Named senior engineers on call for Druid query and broker incidents

Overview

A retail analytics provider exposed Druid-backed dashboards to its own customers, so query latency was a product characteristic rather than an internal concern. Broker timeouts appeared unpredictably and defied the team's attempts to reproduce them. AceMQ took over support with named senior engineers and direct escalation.

Challenge

Druid query latency depends on how many segments a query touches, how well those segments are sized, whether results come from cache, and whether historical processing threads are contended. The timeouts correlated with datasources whose segments were both too numerous and too small, which meant per-segment overhead dominated actual work. Cache configuration was also negating itself on the highest-volume query patterns.

Environment

Apache Druid on AWS serving customer-facing analytics dashboards with published latency expectations.

Approach

AceMQ instrumented per-query segment counts and scan times to distinguish queries that were genuinely expensive from queries paying overhead on fragmented segments. Support then covered both the operational response to timeouts and the structural work needed to stop generating them.

Solution

• Instrumented per-query segment counts and scan times to separate real work from per-segment overhead
• Identified datasources with fragmented segments and scheduled compaction to bring them into the target size range
• Retuned historical processing thread counts and buffer sizing against the underlying instance types
• Corrected broker and historical cache configuration that had been negating itself on high-volume patterns
• Established query timeout, priority, and laning policy so heavy ad-hoc queries cannot starve dashboards
• Provided 24/7 escalation to senior Druid engineers with no tier-1 triage in the path

Outcome

Technologies