Overview
A telecommunications operator's Druid cluster had accumulated years of network telemetry with no compaction policy. Segment counts had grown into the millions, coordinator work grew with them, and every query paid overhead proportional to segment count rather than data volume. AceMQ designed a granularity and compaction strategy.
Challenge
Streaming ingestion naturally produces small segments sized to the ingestion window, not to what queries want. Without compaction, those segments persist indefinitely. The redesign had to pick granularities matched to real query time ranges, define secondary partitioning that improved pruning, and schedule compaction so it did not compete with ingestion for task capacity.
Environment
On-premises Apache Druid cluster storing multi-year network performance and subscriber telemetry.
Approach
AceMQ analyzed the distribution of query time ranges per datasource to pick segment granularities that matched how the data was actually queried, then designed auto-compaction with explicit task capacity reservations so it would run continuously without starving ingestion.
Solution
- Analyzed query time range distribution per datasource to select segment granularity from evidence rather than convention
- Defined secondary partitioning on the dimensions queries filter on most, improving pruning within segments
- Configured auto-compaction with target segment sizes and explicit reserved task capacity
- Scheduled compaction of the historical backlog in bounded phases to avoid overwhelming the cluster
- Set rollup and metric aggregation policy per datasource to cut segment size where query granularity allows
- Established segment count and average segment size per datasource as tracked operational metrics
Outcome
Segment counts fell dramatically as the backlog compacted, and auto-compaction now holds them steady as new data arrives. Coordinator load dropped and query overhead scales with data volume rather than segment count.
Technologies
Apache Druid
Kafka
Linux
Prometheus
S3
Kubernetes
Grafana
