Stopping WiredTiger eviction thrash before it stalls the application tier
MongoDB
Kubernetes
PrometheusLinuxEviction returned to background threads and write latency came back into its normal range, removing the client timeouts entirely. The customer now has an early-warning signal for cache pressure and a …
Overview
A payments platform began seeing multi-second stalls on writes that had previously completed in single-digit milliseconds. Server metrics showed the WiredTiger cache pinned at the eviction trigger with application threads being recruited to evict pages. AceMQ engineers were engaged under the 15-minute emergency SLA and worked the incident with the customer's DBA team.
Challenge
The working set had grown past the WiredTiger cache allocation, so read misses forced dirty page eviction on the hot path. Once application threads started participating in eviction, latency spiked non-linearly and connection pools saturated upstream. The team had been treating it as a network problem because the symptom surfaced as client-side timeouts rather than slow queries.
Environment
MongoDB replica sets on bare metal, backing a high-volume transaction authorization service.
Approach
AceMQ correlated serverStatus eviction counters, cache dirty percentage, and checkpoint duration against the application latency histogram to confirm eviction as the root cause rather than disk or network. From there the work split into immediate pressure relief and durable fixes to the index and document footprint so the working set fit back inside cache.
Solution
- 1Correlated WiredTiger eviction counters, cache dirty ratio, and checkpoint duration with client-side latency to isolate the true bottleneck
- 2Retuned cacheSizeGB against actual host memory and container limits instead of the default half-of-RAM heuristic
- 3Identified and dropped redundant compound indexes that were consuming cache without serving any query shape
- 4Reworked the two heaviest collections to reduce document size and eliminate unbounded array growth
- 5Added eviction pressure, cache dirty percentage, and checkpoint duration to the alerting baseline so the condition is caught before it reaches application threads
Outcome
Eviction returned to background threads and write latency came back into its normal range, removing the client timeouts entirely. The customer now has an early-warning signal for cache pressure and a documented sizing model for future data growth.
Technologies
MongoDBKubernetesPrometheusLinuxRelated Use Cases
MongoDB Replica Set and Oplog Window Support
Ongoing support for replica sets where a short oplog window was forcing repeated full initial syncs of secondaries during nightly batch loads.
MongoDB Deployment Health Assessment
Structured review of schema design, index efficiency, replica set topology, and backup recoverability ahead of a major workload increase.
Facing a MongoDB Production Issue?
AceMQ's senior MongoDB engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.