Overview
An advertising exchange depended on Druid for near-real-time bid analytics. Kafka supervisor lag grew from seconds to hours during peak traffic, so the dashboards traders relied on showed stale data exactly when it mattered. AceMQ engaged under the emergency SLA.
Challenge
The lag had several compounding sources. Middle Manager task slots were exhausted because ingestion tasks were running long and holding slots that pending tasks needed. Segments produced by those tasks were oversized, so handoff to deep storage and loading onto historicals was slow. Failed tasks were being retried into the same saturated slot pool.
Environment
Apache Druid on Kubernetes ingesting bid and impression streams from Kafka for real-time analytics.
Approach
AceMQ traced the ingestion pipeline end to end — supervisor, task assignment, segment publishing, handoff, and historical load — to find where time was actually accumulating rather than tuning the supervisor in isolation. Capacity and task shape were corrected together, since fixing one without the other simply relocates the queue.
Solution
- Traced the full ingestion path from supervisor through handoff and historical load to locate the real delay
- Rebalanced Middle Manager worker capacity and task slot allocation against actual concurrent task demand
- Adjusted task duration and segment granularity so tasks released slots on a predictable cadence
- Corrected segment sizing so handoff and historical loading completed within the ingestion window
- Separated streaming ingestion capacity from batch and compaction task capacity so they stopped competing
- Instrumented supervisor lag, task slot utilization, and handoff duration as distinct alertable signals
Outcome
Ingestion lag returned to the seconds range and held there through peak traffic. Trading dashboards now reflect current data during the periods when they are actually used.
Technologies
Apache Druid
Kafka
Kubernetes
Prometheus
Linux
Grafana
S3
