Stop sizing local NVMe for data you read once a quarter
Apache Kafka
Kubernetes
Apache SparkCustomers extend retention by an order of magnitude without a proportional increase in local storage, and the cost model reflects measured read behavior. Recovery procedures are validated before they …
Overview
Tiered storage moves closed log segments to object storage while keeping the recent window on local disk, which decouples retention from the size of the NVMe you provisioned. AceMQ assesses whether the read patterns justify it and designs the configuration and sizing around measured behavior.
Challenge
Teams facing a multi-month or multi-year retention requirement either over-provision local disk across every broker or truncate retention and lose the ability to replay. Tiered storage solves this, but only if the local cache window matches how consumers actually read — a rebuild job that replays from the beginning of a topic pulls large volumes from object storage, and the resulting request and egress costs surprise people who did not model them.
Environment
Redpanda clusters on Kubernetes or cloud instances with local NVMe and S3-compatible object storage, serving both streaming consumers and periodic historical replay.
Approach
AceMQ profiles consumer read offsets over time to measure how far back consumers actually reach and how often, which determines the local cache window. Retention is then split into a hot local tier and a cold object tier per topic, and remote read cost is modeled against the real replay patterns rather than a worst case.
Solution
- 1Consumer read-offset profiling to measure the actual recency window and the frequency of deep historical replay
- 2Per-topic retention split between local hot storage and remote object storage tiers
- 3Local disk and cache sizing derived from the measured hot window plus headroom for consumer lag events
- 4Object storage request, egress, and lifecycle cost modeling against real replay patterns rather than worst case
- 5Remote read performance validation for the batch and analytics jobs that replay from the cold tier
- 6Recovery and topic recreation procedures validated against the remote tier, including cluster rebuild from object storage
Outcome
Customers extend retention by an order of magnitude without a proportional increase in local storage, and the cost model reflects measured read behavior. Recovery procedures are validated before they are needed rather than during an incident.
Technologies
Apache KafkaKubernetesApache SparkRelated Use Cases
Redpanda Migration from Apache Kafka
Migrating from Kafka to Redpanda with client compatibility testing, ACL and schema registry translation, and a staged cutover per topic.
Redpanda Production Support
Named-engineer 24/7 support for Redpanda clusters covering node recovery, consumer lag incidents, upgrades, and client-side failures.
Ready for a Redpanda Health Check?
AceMQ's senior Redpanda engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.