Overview
An energy utility kept seven years of meter and grid telemetry in Druid, with every segment held on identical historical nodes at the same replication factor regardless of age. Infrastructure cost scaled linearly with retention while query volume against older data was negligible. AceMQ assessed the tiering options.
Challenge
Tiering only works if you know the real access distribution, and assumptions about it are usually wrong — regulatory queries against old data are rare but must still succeed within a defined time. The assessment had to establish actual access patterns by data age, then determine which tiering configurations preserved required query behavior at lower cost.
Environment
Apache Druid across on-premises hardware and cloud object storage, retaining multi-year grid and meter telemetry.
Approach
AceMQ analyzed query logs to build an access distribution by segment age rather than relying on stated expectations, then modeled tiering configurations against both cost and the query behavior each would produce, including cold-tier response times for regulatory access.
Solution
- Built an access distribution by segment age from actual query logs rather than stated assumptions
- Modeled hot, warm, and cold historical tiers with load rules mapped to the measured access pattern
- Evaluated reduced replication factors on cold tiers against availability and recovery requirements
- Assessed query-from-deep-storage as an option for the coldest data and measured its response characteristics
- Reviewed retention and drop rules against regulatory retention obligations
- Delivered a cost and performance model for each tiering option with the trade-offs stated explicitly
Outcome
The utility gained a tiering plan grounded in measured access rather than assumption, with a clear path to reducing historical node footprint while keeping regulatory queries within their required response window. Trade-offs were stated up front rather than discovered after implementation.
Technologies
Apache Druid
S3
Kubernetes
Prometheus
Kafka
Linux
Grafana
PostgreSQL
Apache Spark
