Finding the latency spike that only shows up in the p99.9
RedisLinux
Kubernetes
PrometheusThe periodic tail spikes disappeared and p99.9 latency came back within the platform's budget. Auction timeouts attributable to the cache tier stopped.
Overview
A bidding platform operated under a hard per-request latency budget and was losing auctions to timeouts on a periodic cycle. Average Redis latency looked fine; the p99.9 showed spikes of several hundred milliseconds arriving at regular intervals. AceMQ was engaged under the emergency SLA.
Challenge
The spike interval matched the RDB snapshot schedule. Forking a large Redis process copies page tables, and with transparent huge pages enabled the copy-on-write cost after the fork was multiplied — every write to a shared huge page forced a two-megabyte copy instead of a four-kilobyte one. The effect was invisible in average latency and only appeared in the tail, where this workload lived.
Environment
Redis on bare-metal Linux hosts serving real-time bidding decisions under a strict latency budget.
Approach
AceMQ correlated the spike timestamps against fork duration from the Redis logs and against kernel memory statistics to confirm the mechanism rather than infer it. Fixes addressed the host configuration, the persistence strategy, and the memory footprint that made each fork expensive.
Solution
- 1Correlated tail latency spikes against logged fork duration and kernel memory statistics to confirm the mechanism
- 2Disabled transparent huge pages on the Redis hosts and validated the change against fork cost
- 3Moved snapshot responsibility to replicas so the primary no longer forks on the serving path
- 4Reviewed persistence strategy and rewrite thresholds against the platform's real durability requirements
- 5Reduced per-instance memory footprint through key structure changes so any remaining fork is cheaper
- 6Instrumented tail latency percentiles and fork duration as first-class monitored signals
Outcome
The periodic tail spikes disappeared and p99.9 latency came back within the platform's budget. Auction timeouts attributable to the cache tier stopped.
Technologies
RedisLinuxKubernetesPrometheusRelated Use Cases
Redis Memory and Eviction Policy Support
Ongoing support for instances where the eviction policy did not match how the keyspace was used, causing session data to be evicted under memory pressure.
Redis Cluster Mode Migration Consulting
Planning a migration from vertically scaled standalone Redis to Cluster mode, including hash tag design and remediation of cross-slot operations.
Facing a Redis Production Issue?
AceMQ's senior Redis engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.