When p99 produce latency jumps, the leadership distribution is the first place to look

Overview

Redpanda's per-partition Raft groups make latency sensitive to how partition leadership is distributed. When leaders concentrate on a subset of brokers, those nodes saturate while the rest of the cluster is idle, and tail latency rises well before average latency shows anything. AceMQ diagnoses and rebalances.

Challenge

Latency spikes appear after a rolling restart, a node replacement, or organic partition growth. Leadership ends up unevenly distributed, so a few brokers carry disproportionate write load. Underneath, the storage class may not deliver the sustained write throughput the cluster assumed, Kubernetes CPU limits may be throttling the reactor threads, or a topic's partition count may be too low to spread load regardless of leadership. The symptom is the same in every case.

Environment

Redpanda clusters on Kubernetes or cloud instances with local NVMe or network-attached storage, serving latency-sensitive producers.

Approach

AceMQ correlates per-broker leadership counts, disk write latency, and CPU throttling metrics against the latency percentiles to identify which layer is actually saturated. Leadership is rebalanced and partition assignment corrected first, then storage class and resource limits are addressed where the hardware is genuinely the constraint.

Solution

• Per-broker Raft leadership count and partition distribution analysis correlated with produce latency percentiles
• Leadership rebalancing and partition reassignment to spread write load evenly across the cluster
• Disk write latency and sustained throughput validation against the storage class actually provisioned
• Kubernetes CPU limit and reactor thread review to eliminate throttling of the seastar shard threads
• Partition count review for topics whose parallelism is too low for their throughput regardless of leadership placement
• Producer batching, linger, and acknowledgment settings tuned against the durability requirement rather than defaults

Outcome

Technologies