Stop sizing local NVMe for data you read once a quarter

Overview

Tiered storage moves closed log segments to object storage while keeping the recent window on local disk, which decouples retention from the size of the NVMe you provisioned. AceMQ assesses whether the read patterns justify it and designs the configuration and sizing around measured behavior.

Challenge

Teams facing a multi-month or multi-year retention requirement either over-provision local disk across every broker or truncate retention and lose the ability to replay. Tiered storage solves this, but only if the local cache window matches how consumers actually read — a rebuild job that replays from the beginning of a topic pulls large volumes from object storage, and the resulting request and egress costs surprise people who did not model them.

Environment

Redpanda clusters on Kubernetes or cloud instances with local NVMe and S3-compatible object storage, serving both streaming consumers and periodic historical replay.

Approach

AceMQ profiles consumer read offsets over time to measure how far back consumers actually reach and how often, which determines the local cache window. Retention is then split into a hot local tier and a cold object tier per topic, and remote read cost is modeled against the real replay patterns rather than a worst case.

Solution

• Consumer read-offset profiling to measure the actual recency window and the frequency of deep historical replay
• Per-topic retention split between local hot storage and remote object storage tiers
• Local disk and cache sizing derived from the measured hot window plus headroom for consumer lag events
• Object storage request, egress, and lifecycle cost modeling against real replay patterns rather than worst case
• Remote read performance validation for the batch and analytics jobs that replay from the cold tier
• Recovery and topic recreation procedures validated against the remote tier, including cluster rebuild from object storage

Outcome

Technologies