Memory that only ever went up
RedisRedis SentinelContainersObservabilityMemory growth reduced and the production system stabilised. No critical failures were reported in the period following the engagement, and the platform kept running through the remediation rather than…
Overview
A SaaS platform running real-time screen sharing had a Redis cluster whose memory climbed steadily between restarts and never came back down. Restarting cleared it, which made the problem easy to live with and easy to misdiagnose. AceMQ ran the engagement as six live working sessions totalling fourteen hours, with the customer's engineers in the room rather than as a written report delivered weeks later.
Challenge
Two separate problems presented as one symptom. The memory ceiling had been misconfigured at 7GB, so the instance was not evicting when it should have been, and Redis Search was degrading under a high-frequency write pattern the index had not been designed for. Either alone was survivable. Together they produced growth that looked like a classic leak and resisted the usual eviction-policy fixes.
Environment
Containerised Redis with Sentinel-managed failover, serving session state for a real-time collaboration product.
Approach
The sessions were deliberately interactive: reproduce, measure, change one variable, measure again. That format let the team separate the ceiling misconfiguration from the Redis Search behaviour instead of applying both fixes at once and learning nothing.
Solution
- 1Corrected the memory ceiling so the eviction policy could actually engage at the intended threshold
- 2Identified Redis Search degradation under high-frequency writes as a distinct cause rather than a symptom of the leak
- 3Introduced staggered container rotation to reclaim memory progressively, avoiding a simultaneous restart of the cluster
- 4Reviewed Sentinel configuration so failover behaviour during rotation was predictable
- 5Added the memory and index metrics that would have surfaced both causes earlier
- 6Left the customer's engineers able to run the rotation themselves rather than depending on AceMQ to do it
Outcome
Memory growth reduced and the production system stabilised. No critical failures were reported in the period following the engagement, and the platform kept running through the remediation rather than taking a maintenance window.
Technologies
RedisRedis SentinelContainersObservabilityRelated Use Cases
Redis Latency Spike Remediation from Fork Stalls
Eliminating periodic multi-hundred-millisecond latency spikes traced to RDB snapshot fork stalls amplified by transparent huge pages.
Redis Memory and Eviction Policy Support
Ongoing support for instances where the eviction policy did not match how the keyspace was used, causing session data to be evicted under memory pressure.
Need Expert Redis Support?
AceMQ's senior Redis engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.