Compaction policy that keeps segment counts bounded over years

Overview

A telecommunications operator's Druid cluster had accumulated years of network telemetry with no compaction policy. Segment counts had grown into the millions, coordinator work grew with them, and every query paid overhead proportional to segment count rather than data volume. AceMQ designed a granularity and compaction strategy.

Challenge

Streaming ingestion naturally produces small segments sized to the ingestion window, not to what queries want. Without compaction, those segments persist indefinitely. The redesign had to pick granularities matched to real query time ranges, define secondary partitioning that improved pruning, and schedule compaction so it did not compete with ingestion for task capacity.

Environment

On-premises Apache Druid cluster storing multi-year network performance and subscriber telemetry.

Approach

AceMQ analyzed the distribution of query time ranges per datasource to pick segment granularities that matched how the data was actually queried, then designed auto-compaction with explicit task capacity reservations so it would run continuously without starving ingestion.

Solution

• Analyzed query time range distribution per datasource to select segment granularity from evidence rather than convention
• Defined secondary partitioning on the dimensions queries filter on most, improving pruning within segments
• Configured auto-compaction with target segment sizes and explicit reserved task capacity
• Scheduled compaction of the historical backlog in bounded phases to avoid overwhelming the cluster
• Set rollup and metric aggregation policy per datasource to cut segment size where query granularity allows
• Established segment count and average segment size per datasource as tracked operational metrics

Outcome

Technologies