Getting real-time ingestion back inside its latency budget

Overview

An advertising exchange depended on Druid for near-real-time bid analytics. Kafka supervisor lag grew from seconds to hours during peak traffic, so the dashboards traders relied on showed stale data exactly when it mattered. AceMQ engaged under the emergency SLA.

Challenge

The lag had several compounding sources. Middle Manager task slots were exhausted because ingestion tasks were running long and holding slots that pending tasks needed. Segments produced by those tasks were oversized, so handoff to deep storage and loading onto historicals was slow. Failed tasks were being retried into the same saturated slot pool.

Environment

Apache Druid on Kubernetes ingesting bid and impression streams from Kafka for real-time analytics.

Approach

AceMQ traced the ingestion pipeline end to end — supervisor, task assignment, segment publishing, handoff, and historical load — to find where time was actually accumulating rather than tuning the supervisor in isolation. Capacity and task shape were corrected together, since fixing one without the other simply relocates the queue.

Solution

• Traced the full ingestion path from supervisor through handoff and historical load to locate the real delay
• Rebalanced Middle Manager worker capacity and task slot allocation against actual concurrent task demand
• Adjusted task duration and segment granularity so tasks released slots on a predictable cadence
• Corrected segment sizing so handoff and historical loading completed within the ingestion window
• Separated streaming ingestion capacity from batch and compaction task capacity so they stopped competing
• Instrumented supervisor lag, task slot utilization, and handoff duration as distinct alertable signals

Outcome

Technologies