Memory growth that looked like a workload problem turned out to be an indexing bug
Redis
KubernetesMemory growth stopped being an incident. The cluster held steady between maintenance windows, the team gained a documented mitigation they could run themselves, and the Valkey migration proceeded as a…
Overview
A collaboration platform ran Redis as the session and search tier behind a customer-facing product. Memory climbed steadily between restarts until the cluster crashed, and the pattern had survived several rounds of resizing.
Challenge
Adding memory did not help, because the growth was not proportional to traffic. The cluster also carried a memory ceiling that had been set without units, so the intended limit was never actually applied. Restarts recovered memory but dropped sessions.
Environment
Containerized Redis with replicas and Sentinel-based failover, running in a managed cloud environment behind a high-frequency write workload.
Approach
AceMQ engineers reproduced the growth in a CI cluster, isolated it to the full-text index by dropping and rebuilding the index and watching memory return, and confirmed the behavior matched a known defect in the search module version in use. A staggered restart of replicas before the master gave the team a safe interim mitigation while the upgrade path was evaluated.
Solution
- 1Reproduced memory growth in a non-production cluster to isolate the cause
- 2Traced the leak to the full-text search module rather than the application workload
- 3Corrected a memory ceiling that had been configured without units
- 4Set an explicit eviction policy so the cluster degrades instead of crashing
- 5Introduced staggered replica-then-master restarts to avoid failover risk
- 6Evaluated Valkey as the supported long-term path, with feature-flagged rollout
Outcome
Memory growth stopped being an incident. The cluster held steady between maintenance windows, the team gained a documented mitigation they could run themselves, and the Valkey migration proceeded as a planned change rather than an emergency.
Technologies
RedisKubernetesRelated Use Cases
Redis Cluster Health and Architecture Assessment
AceMQ's Redis Health and Architecture Assessment identifies cluster vulnerabilities, performance bottlenecks, and optimization opportunities before they become production incidents — delivering a prioritized remediation roadmap.
Docker OOMKilled Container Remediation
Resolving containers repeatedly OOMKilled because the JVM and Node runtimes inside them were sizing heap against host memory rather than the cgroup limit.
Redis Latency Spike Remediation from Fork Stalls
Eliminating periodic multi-hundred-millisecond latency spikes traced to RDB snapshot fork stalls amplified by transparent huge pages.
Apigee Quota and Spike Arrest Behavior Remediation
Correcting Apigee quota and spike arrest configuration that was rejecting legitimate traffic while letting genuine bursts through to backends.
Kong Rate Limiting Consistency Support
Fixing rate limits that allowed several times the configured quota because the plugin was using the local counter policy across a multi-node gateway.
Redis Memory and Eviction Policy Support
Ongoing support for instances where the eviction policy did not match how the keyspace was used, causing session data to be evicted under memory pressure.
Facing a Valkey Production Issue?
AceMQ's senior Valkey engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.