Stopping WiredTiger eviction thrash before it stalls the application tier

Overview

A payments platform began seeing multi-second stalls on writes that had previously completed in single-digit milliseconds. Server metrics showed the WiredTiger cache pinned at the eviction trigger with application threads being recruited to evict pages. AceMQ engineers were engaged under the 15-minute emergency SLA and worked the incident with the customer's DBA team.

Challenge

The working set had grown past the WiredTiger cache allocation, so read misses forced dirty page eviction on the hot path. Once application threads started participating in eviction, latency spiked non-linearly and connection pools saturated upstream. The team had been treating it as a network problem because the symptom surfaced as client-side timeouts rather than slow queries.

Environment

MongoDB replica sets on bare metal, backing a high-volume transaction authorization service.

Approach

AceMQ correlated serverStatus eviction counters, cache dirty percentage, and checkpoint duration against the application latency histogram to confirm eviction as the root cause rather than disk or network. From there the work split into immediate pressure relief and durable fixes to the index and document footprint so the working set fit back inside cache.

Solution

• Correlated WiredTiger eviction counters, cache dirty ratio, and checkpoint duration with client-side latency to isolate the true bottleneck
• Retuned cacheSizeGB against actual host memory and container limits instead of the default half-of-RAM heuristic
• Identified and dropped redundant compound indexes that were consuming cache without serving any query shape
• Reworked the two heaviest collections to reduce document size and eliminate unbounded array growth
• Added eviction pressure, cache dirty percentage, and checkpoint duration to the alerting baseline so the condition is caught before it reaches application threads

Outcome

Technologies