When p99 produce latency jumps, the leadership distribution is the first place to look
Tail latency comes back into range and stays there through rolling restarts, because leadership rebalancing is applied as an operational practice rather than a one-time fix. Where storage was the real…
Overview
Redpanda's per-partition Raft groups make latency sensitive to how partition leadership is distributed. When leaders concentrate on a subset of brokers, those nodes saturate while the rest of the cluster is idle, and tail latency rises well before average latency shows anything. AceMQ diagnoses and rebalances.
Challenge
Latency spikes appear after a rolling restart, a node replacement, or organic partition growth. Leadership ends up unevenly distributed, so a few brokers carry disproportionate write load. Underneath, the storage class may not deliver the sustained write throughput the cluster assumed, Kubernetes CPU limits may be throttling the reactor threads, or a topic's partition count may be too low to spread load regardless of leadership. The symptom is the same in every case.
Environment
Redpanda clusters on Kubernetes or cloud instances with local NVMe or network-attached storage, serving latency-sensitive producers.
Approach
AceMQ correlates per-broker leadership counts, disk write latency, and CPU throttling metrics against the latency percentiles to identify which layer is actually saturated. Leadership is rebalanced and partition assignment corrected first, then storage class and resource limits are addressed where the hardware is genuinely the constraint.
Solution
- 1Per-broker Raft leadership count and partition distribution analysis correlated with produce latency percentiles
- 2Leadership rebalancing and partition reassignment to spread write load evenly across the cluster
- 3Disk write latency and sustained throughput validation against the storage class actually provisioned
- 4Kubernetes CPU limit and reactor thread review to eliminate throttling of the seastar shard threads
- 5Partition count review for topics whose parallelism is too low for their throughput regardless of leadership placement
- 6Producer batching, linger, and acknowledgment settings tuned against the durability requirement rather than defaults
Outcome
Tail latency comes back into range and stays there through rolling restarts, because leadership rebalancing is applied as an operational practice rather than a one-time fix. Where storage was the real constraint, the sizing recommendation is backed by measured throughput.
Technologies
Related Use Cases
Redpanda Production Support
Named-engineer 24/7 support for Redpanda clusters covering node recovery, consumer lag incidents, upgrades, and client-side failures.
Redpanda Tiered Storage Assessment
Designing tiered storage for long retention so historical data lives in object storage without local disk dictating how long you can keep it.
Facing a Redpanda Production Issue?
AceMQ's senior Redpanda engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.