Surviving the moment the cache goes cold

Overview

A digital media platform repeatedly took database outages whenever a Memcached node was replaced or a large batch of keys expired together. AceMQ provides continuous support covering the cache tier and the application patterns around it.

Challenge

Losing a node in a consistent-hashing ring remaps its share of keys, and every remapped key misses at once. With no protection against concurrent regeneration, thousands of requests would recompute the same expensive value simultaneously and saturate the database. The same happened at expiry cliffs, because keys populated together during a deployment shared an identical TTL and therefore expired together. Routine node maintenance had become a change-approval problem.

Environment

Memcached tier on AWS fronting a relational database, application servers on Kubernetes with frequent deployments.

Approach

Support here spans the cache and the application patterns that use it, since stampede protection is an application concern that the cache alone cannot solve. We work with the customer's engineers on request coalescing and TTL jitter, and make node topology changes a controlled operation rather than an event.

Solution

• 24/7 support with a 15-minute emergency SLA and named senior engineers who know the cache topology and its consumers
• Request coalescing added at the application layer so only one caller regenerates a missing value while others wait
• TTL jitter applied so keys populated together do not expire together and produce an expiry cliff
• Consistent hashing configuration reviewed so node changes remap the minimum possible key share
• Controlled node addition and removal procedure with pre-warming, replacing unplanned replacement
• Hit rate, eviction, and get-miss monitoring per node so a degrading node is caught before it is lost

Outcome

Technologies