Evictions climbing while the cache reports memory to spare

Overview

A commerce platform's cache hit rate collapsed after a change to their serialized object format, and evictions climbed sharply even though the servers reported unused memory. The database took the resulting load and page latency followed.

Challenge

Memcached assigns memory to slab classes by item size, and once a page is assigned to a class it stays there by default. The serialization change shifted the typical item into a different size class than the one holding nearly all the allocated pages. The new class filled and began evicting almost immediately while the old class sat mostly empty — the classic calcification signature. Total memory looked fine on every dashboard, which is why the team had ruled out capacity as a cause.

Environment

Memcached tier fronting a relational database for session and object caching, application servers on Kubernetes.

Approach

We read the per-slab-class statistics rather than the aggregate memory numbers, which made the imbalance obvious within minutes. Immediate relief came from enabling slab rebalancing so pages could migrate between classes. The durable fix was to size items deliberately and to monitor the metrics that actually reveal this failure mode.

Solution

• Per-slab-class statistics analyzed to expose the eviction-versus-free-memory contradiction the aggregate metrics hid
• Automatic slab rebalancing enabled so pages migrate between size classes as the workload shifts
• Growth factor and chunk sizing reviewed against the real item size distribution rather than defaults
• Serialized object sizes reduced at the application layer so items sit predictably within a class
• Per-class eviction and page count monitoring added, replacing the aggregate memory graph that concealed the problem
• Restart and warm-up procedure defined so a cache restart does not stampede the database

Outcome

Technologies