Matching storage tier to how often the data is actually read
Kubernetes
PrometheusThe utility gained a tiering plan grounded in measured access rather than assumption, with a clear path to reducing historical node footprint while keeping regulatory queries within their required res…
Overview
An energy utility kept seven years of meter and grid telemetry in Druid, with every segment held on identical historical nodes at the same replication factor regardless of age. Infrastructure cost scaled linearly with retention while query volume against older data was negligible. AceMQ assessed the tiering options.
Challenge
Tiering only works if you know the real access distribution, and assumptions about it are usually wrong — regulatory queries against old data are rare but must still succeed within a defined time. The assessment had to establish actual access patterns by data age, then determine which tiering configurations preserved required query behavior at lower cost.
Environment
Apache Druid across on-premises hardware and cloud object storage, retaining multi-year grid and meter telemetry.
Approach
AceMQ analyzed query logs to build an access distribution by segment age rather than relying on stated expectations, then modeled tiering configurations against both cost and the query behavior each would produce, including cold-tier response times for regulatory access.
Solution
- 1Built an access distribution by segment age from actual query logs rather than stated assumptions
- 2Modeled hot, warm, and cold historical tiers with load rules mapped to the measured access pattern
- 3Evaluated reduced replication factors on cold tiers against availability and recovery requirements
- 4Assessed query-from-deep-storage as an option for the coldest data and measured its response characteristics
- 5Reviewed retention and drop rules against regulatory retention obligations
- 6Delivered a cost and performance model for each tiering option with the trade-offs stated explicitly
Outcome
The utility gained a tiering plan grounded in measured access rather than assumption, with a clear path to reducing historical node footprint while keeping regulatory queries within their required response window. Trade-offs were stated up front rather than discovered after implementation.
Technologies
KubernetesPrometheusRelated Use Cases
Apache Druid Segment Granularity and Compaction Design
Redesigning segment granularity, partitioning, and auto-compaction policy so segment counts stay bounded as historical data accumulates.
Apache Druid Query Performance Support
Ongoing support for broker timeouts and unpredictable query latency driven by segment sizing, cache behavior, and processing thread contention.
Ready for a Apache Druid Health Check?
AceMQ's senior Apache Druid engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.