Getting real-time ingestion back inside its latency budget
Kubernetes
PrometheusIngestion lag returned to the seconds range and held there through peak traffic. Trading dashboards now reflect current data during the periods when they are actually used.
Overview
An advertising exchange depended on Druid for near-real-time bid analytics. Kafka supervisor lag grew from seconds to hours during peak traffic, so the dashboards traders relied on showed stale data exactly when it mattered. AceMQ engaged under the emergency SLA.
Challenge
The lag had several compounding sources. Middle Manager task slots were exhausted because ingestion tasks were running long and holding slots that pending tasks needed. Segments produced by those tasks were oversized, so handoff to deep storage and loading onto historicals was slow. Failed tasks were being retried into the same saturated slot pool.
Environment
Apache Druid on Kubernetes ingesting bid and impression streams from Kafka for real-time analytics.
Approach
AceMQ traced the ingestion pipeline end to end — supervisor, task assignment, segment publishing, handoff, and historical load — to find where time was actually accumulating rather than tuning the supervisor in isolation. Capacity and task shape were corrected together, since fixing one without the other simply relocates the queue.
Solution
- 1Traced the full ingestion path from supervisor through handoff and historical load to locate the real delay
- 2Rebalanced Middle Manager worker capacity and task slot allocation against actual concurrent task demand
- 3Adjusted task duration and segment granularity so tasks released slots on a predictable cadence
- 4Corrected segment sizing so handoff and historical loading completed within the ingestion window
- 5Separated streaming ingestion capacity from batch and compaction task capacity so they stopped competing
- 6Instrumented supervisor lag, task slot utilization, and handoff duration as distinct alertable signals
Outcome
Ingestion lag returned to the seconds range and held there through peak traffic. Trading dashboards now reflect current data during the periods when they are actually used.
Technologies
KubernetesPrometheusRelated Use Cases
Apache Druid Query Performance Support
Ongoing support for broker timeouts and unpredictable query latency driven by segment sizing, cache behavior, and processing thread contention.
Apache Druid Segment Granularity and Compaction Design
Redesigning segment granularity, partitioning, and auto-compaction policy so segment counts stay bounded as historical data accumulates.
Facing a Apache Druid Production Issue?
AceMQ's senior Apache Druid engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.