Compaction policy that keeps segment counts bounded over years
PrometheusSegment counts fell dramatically as the backlog compacted, and auto-compaction now holds them steady as new data arrives. Coordinator load dropped and query overhead scales with data volume rather tha…
Overview
A telecommunications operator's Druid cluster had accumulated years of network telemetry with no compaction policy. Segment counts had grown into the millions, coordinator work grew with them, and every query paid overhead proportional to segment count rather than data volume. AceMQ designed a granularity and compaction strategy.
Challenge
Streaming ingestion naturally produces small segments sized to the ingestion window, not to what queries want. Without compaction, those segments persist indefinitely. The redesign had to pick granularities matched to real query time ranges, define secondary partitioning that improved pruning, and schedule compaction so it did not compete with ingestion for task capacity.
Environment
On-premises Apache Druid cluster storing multi-year network performance and subscriber telemetry.
Approach
AceMQ analyzed the distribution of query time ranges per datasource to pick segment granularities that matched how the data was actually queried, then designed auto-compaction with explicit task capacity reservations so it would run continuously without starving ingestion.
Solution
- 1Analyzed query time range distribution per datasource to select segment granularity from evidence rather than convention
- 2Defined secondary partitioning on the dimensions queries filter on most, improving pruning within segments
- 3Configured auto-compaction with target segment sizes and explicit reserved task capacity
- 4Scheduled compaction of the historical backlog in bounded phases to avoid overwhelming the cluster
- 5Set rollup and metric aggregation policy per datasource to cut segment size where query granularity allows
- 6Established segment count and average segment size per datasource as tracked operational metrics
Outcome
Segment counts fell dramatically as the backlog compacted, and auto-compaction now holds them steady as new data arrives. Coordinator load dropped and query overhead scales with data volume rather than segment count.
Technologies
PrometheusRelated Use Cases
Apache Druid Tiered Historical Architecture Assessment
Assessment of historical tiering, retention rules, and replication factors to align infrastructure cost with how data is actually queried over time.
Apache Druid Streaming Ingestion Lag Remediation
Resolving Kafka supervisor lag caused by task slot exhaustion, oversized ingestion tasks, and handoff failures to deep storage.
Need Apache Druid Architecture Guidance?
AceMQ's senior Apache Druid engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.