Matching storage tier to how often the data is actually read

Overview

An energy utility kept seven years of meter and grid telemetry in Druid, with every segment held on identical historical nodes at the same replication factor regardless of age. Infrastructure cost scaled linearly with retention while query volume against older data was negligible. AceMQ assessed the tiering options.

Challenge

Tiering only works if you know the real access distribution, and assumptions about it are usually wrong — regulatory queries against old data are rare but must still succeed within a defined time. The assessment had to establish actual access patterns by data age, then determine which tiering configurations preserved required query behavior at lower cost.

Environment

Apache Druid across on-premises hardware and cloud object storage, retaining multi-year grid and meter telemetry.

Approach

AceMQ analyzed query logs to build an access distribution by segment age rather than relying on stated expectations, then modeled tiering configurations against both cost and the query behavior each would produce, including cold-tier response times for regulatory access.

Solution

• Built an access distribution by segment age from actual query logs rather than stated assumptions
• Modeled hot, warm, and cold historical tiers with load rules mapped to the measured access pattern
• Evaluated reduced replication factors on cold tiers against availability and recovery requirements
• Assessed query-from-deep-storage as an option for the coldest data and measured its response characteristics
• Reviewed retention and drop rules against regulatory retention obligations
• Delivered a cost and performance model for each tiering option with the trade-offs stated explicitly

Outcome

Technologies