Finding the latency spike that only shows up in the p99.9
RedisLinux
Kubernetes
PrometheusThe periodic tail spikes disappeared and p99.9 latency came back within the platform's budget. Auction timeouts attributable to the cache tier stopped.
Overview
A bidding platform operated under a hard per-request latency budget and was losing auctions to timeouts on a periodic cycle. Average Redis latency looked fine; the p99.9 showed spikes of several hundred milliseconds arriving at regular intervals. AceMQ was engaged under the emergency SLA.
Challenge
The spike interval matched the RDB snapshot schedule. Forking a large Redis process copies page tables, and with transparent huge pages enabled the copy-on-write cost after the fork was multiplied — every write to a shared huge page forced a two-megabyte copy instead of a four-kilobyte one. The effect was invisible in average latency and only appeared in the tail, where this workload lived.
Environment
Redis on bare-metal Linux hosts serving real-time bidding decisions under a strict latency budget.
Approach
AceMQ correlated the spike timestamps against fork duration from the Redis logs and against kernel memory statistics to confirm the mechanism rather than infer it. Fixes addressed the host configuration, the persistence strategy, and the memory footprint that made each fork expensive.
Solution
- 1Correlated tail latency spikes against logged fork duration and kernel memory statistics to confirm the mechanism
- 2Disabled transparent huge pages on the Redis hosts and validated the change against fork cost
- 3Moved snapshot responsibility to replicas so the primary no longer forks on the serving path
- 4Reviewed persistence strategy and rewrite thresholds against the platform's real durability requirements
- 5Reduced per-instance memory footprint through key structure changes so any remaining fork is cheaper
- 6Instrumented tail latency percentiles and fork duration as first-class monitored signals
Outcome
The periodic tail spikes disappeared and p99.9 latency came back within the platform's budget. Auction timeouts attributable to the cache tier stopped.
Technologies
RedisLinuxKubernetesPrometheusRelated Use Cases
Redis Memory and Eviction Policy Support
Ongoing support for instances where the eviction policy did not match how the keyspace was used, causing session data to be evicted under memory pressure.
Redis Cluster Mode Migration Consulting
Planning a migration from vertically scaled standalone Redis to Cluster mode, including hash tag design and remediation of cross-slot operations.
Cassandra Tombstone Accumulation and Read Timeout Remediation
Resolving read timeouts caused by tombstone accumulation on queue-like partitions where deletes outpaced compaction and gc_grace_seconds.
MongoDB WiredTiger Cache Eviction Remediation
Diagnosing and resolving application stalls caused by a working set that outgrew the WiredTiger cache, pushing the server into continuous eviction pressure.
Cassandra Cluster Health and Capacity Assessment
Assessment covering topology, replication and consistency configuration, JVM and garbage collection behavior, compaction health, and growth headroom.
Redis to Valkey Migration Under Memory Pressure
A production Redis cluster was crashing under memory growth traced to its full-text index rather than to the workload. AceMQ diagnosed the leak, stabilized the cluster, and mapped the Valkey migration path.
Facing a Redis Production Issue?
AceMQ's senior Redis engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.