Named senior engineers on call for Druid query and broker incidents
Kubernetes
PrometheusBroker timeouts became rare and p99 dashboard latency roughly halved as segment fragmentation was retired. Query laning keeps heavy ad-hoc analysis from affecting the customer-facing path even when it…
Overview
A retail analytics provider exposed Druid-backed dashboards to its own customers, so query latency was a product characteristic rather than an internal concern. Broker timeouts appeared unpredictably and defied the team's attempts to reproduce them. AceMQ took over support with named senior engineers and direct escalation.
Challenge
Druid query latency depends on how many segments a query touches, how well those segments are sized, whether results come from cache, and whether historical processing threads are contended. The timeouts correlated with datasources whose segments were both too numerous and too small, which meant per-segment overhead dominated actual work. Cache configuration was also negating itself on the highest-volume query patterns.
Environment
Apache Druid on AWS serving customer-facing analytics dashboards with published latency expectations.
Approach
AceMQ instrumented per-query segment counts and scan times to distinguish queries that were genuinely expensive from queries paying overhead on fragmented segments. Support then covered both the operational response to timeouts and the structural work needed to stop generating them.
Solution
- 1Instrumented per-query segment counts and scan times to separate real work from per-segment overhead
- 2Identified datasources with fragmented segments and scheduled compaction to bring them into the target size range
- 3Retuned historical processing thread counts and buffer sizing against the underlying instance types
- 4Corrected broker and historical cache configuration that had been negating itself on high-volume patterns
- 5Established query timeout, priority, and laning policy so heavy ad-hoc queries cannot starve dashboards
- 6Provided 24/7 escalation to senior Druid engineers with no tier-1 triage in the path
Outcome
Broker timeouts became rare and p99 dashboard latency roughly halved as segment fragmentation was retired. Query laning keeps heavy ad-hoc analysis from affecting the customer-facing path even when it runs during business hours.
Technologies
KubernetesPrometheusRelated Use Cases
Apache Druid Streaming Ingestion Lag Remediation
Resolving Kafka supervisor lag caused by task slot exhaustion, oversized ingestion tasks, and handoff failures to deep storage.
Apache Druid Tiered Historical Architecture Assessment
Assessment of historical tiering, retention rules, and replication factors to align infrastructure cost with how data is actually queried over time.
Need Expert Apache Druid Support?
AceMQ's senior Apache Druid engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.