Finding the latency spike that only shows up in the p99.9

Overview

A bidding platform operated under a hard per-request latency budget and was losing auctions to timeouts on a periodic cycle. Average Redis latency looked fine; the p99.9 showed spikes of several hundred milliseconds arriving at regular intervals. AceMQ was engaged under the emergency SLA.

Challenge

The spike interval matched the RDB snapshot schedule. Forking a large Redis process copies page tables, and with transparent huge pages enabled the copy-on-write cost after the fork was multiplied — every write to a shared huge page forced a two-megabyte copy instead of a four-kilobyte one. The effect was invisible in average latency and only appeared in the tail, where this workload lived.

Environment

Redis on bare-metal Linux hosts serving real-time bidding decisions under a strict latency budget.

Approach

AceMQ correlated the spike timestamps against fork duration from the Redis logs and against kernel memory statistics to confirm the mechanism rather than infer it. Fixes addressed the host configuration, the persistence strategy, and the memory footprint that made each fork expensive.

Solution

• Correlated tail latency spikes against logged fork duration and kernel memory statistics to confirm the mechanism
• Disabled transparent huge pages on the Redis hosts and validated the change against fork cost
• Moved snapshot responsibility to replicas so the primary no longer forks on the serving path
• Reviewed persistence strategy and rewrite thresholds against the platform's real durability requirements
• Reduced per-instance memory footprint through key structure changes so any remaining fork is cheaper
• Instrumented tail latency percentiles and fork duration as first-class monitored signals

Outcome

Technologies