Establishing what is actually recoverable and what is not
RedisRedis SentinelLinux
PrometheusThe agency received a documented, evidence-backed statement of what each workload survives, replacing an assumption that had never been tested. Two workloads believed to be durable turned out not to b…
Overview
A public sector agency ran Redis for several workloads with different durability needs but a single inherited configuration. Nobody could state which data survived a host failure and which did not. AceMQ assessed the deployment against each workload's real requirements.
Challenge
Redis persistence options trade durability against latency in ways that are easy to misconfigure and hard to notice until a failure. The agency's instances used snapshot intervals that would lose minutes of writes, while some workloads had been assumed durable. Replication and failover had never been tested, so promotion behavior and data loss on failover were unknown.
Environment
On-premises Redis instances with Sentinel-managed failover, supporting caching, queuing, and session workloads.
Approach
AceMQ classified each workload by what loss it could actually tolerate, then compared that against what the current configuration guaranteed. Failover was exercised deliberately, with data loss measured rather than estimated, so the gap between assumption and behavior was documented with evidence.
Solution
- 1Classified each workload by tolerable data loss and compared it against what the configuration actually guarantees
- 2Reviewed snapshot intervals and append-only file fsync policy per instance against those requirements
- 3Exercised Sentinel failover deliberately and measured actual data loss and promotion time
- 4Assessed replication topology, quorum configuration, and split-brain exposure
- 5Verified restore procedures from existing persistence artifacts with timed recovery tests
- 6Delivered a per-workload durability matrix with the configuration changes required to close each gap
Outcome
The agency received a documented, evidence-backed statement of what each workload survives, replacing an assumption that had never been tested. Two workloads believed to be durable turned out not to be, and were reconfigured before a failure demonstrated it.
Technologies
RedisRedis SentinelLinuxPrometheusRelated Use Cases
Redis Cluster Mode Migration Consulting
Planning a migration from vertically scaled standalone Redis to Cluster mode, including hash tag design and remediation of cross-slot operations.
Redis Memory and Eviction Policy Support
Ongoing support for instances where the eviction policy did not match how the keyspace was used, causing session data to be evicted under memory pressure.
Ready for a Redis Health Check?
AceMQ's senior Redis engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.