Evictions climbing while the cache reports memory to spare
KubernetesHit rate returned to its previous level within the hour and evictions on the hot class dropped to near zero. Database read load fell back to normal, and the per-class monitoring has since flagged a se…
Overview
A commerce platform's cache hit rate collapsed after a change to their serialized object format, and evictions climbed sharply even though the servers reported unused memory. The database took the resulting load and page latency followed.
Challenge
Memcached assigns memory to slab classes by item size, and once a page is assigned to a class it stays there by default. The serialization change shifted the typical item into a different size class than the one holding nearly all the allocated pages. The new class filled and began evicting almost immediately while the old class sat mostly empty — the classic calcification signature. Total memory looked fine on every dashboard, which is why the team had ruled out capacity as a cause.
Environment
Memcached tier fronting a relational database for session and object caching, application servers on Kubernetes.
Approach
We read the per-slab-class statistics rather than the aggregate memory numbers, which made the imbalance obvious within minutes. Immediate relief came from enabling slab rebalancing so pages could migrate between classes. The durable fix was to size items deliberately and to monitor the metrics that actually reveal this failure mode.
Solution
- 1Per-slab-class statistics analyzed to expose the eviction-versus-free-memory contradiction the aggregate metrics hid
- 2Automatic slab rebalancing enabled so pages migrate between size classes as the workload shifts
- 3Growth factor and chunk sizing reviewed against the real item size distribution rather than defaults
- 4Serialized object sizes reduced at the application layer so items sit predictably within a class
- 5Per-class eviction and page count monitoring added, replacing the aggregate memory graph that concealed the problem
- 6Restart and warm-up procedure defined so a cache restart does not stampede the database
Outcome
Hit rate returned to its previous level within the hour and evictions on the hot class dropped to near zero. Database read load fell back to normal, and the per-class monitoring has since flagged a second size shift before it caused any impact.
Technologies
KubernetesRelated Use Cases
Memcached Cache Stampede and Eviction Support
Ongoing support for a Memcached tier prone to thundering-herd database load after node changes and cache expiry cliffs.
Memcached Capacity and Hit Rate Assessment
Assessment of Memcached sizing, key distribution, and hit rate to determine whether adding capacity would actually help.
Facing a Memcached Production Issue?
AceMQ's senior Memcached engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.