When p99 produce latency jumps, the leadership distribution is the first place to look
Kubernetes
Apache KafkaTail latency comes back into range and stays there through rolling restarts, because leadership rebalancing is applied as an operational practice rather than a one-time fix. Where storage was the real…
Overview
Redpanda's per-partition Raft groups make latency sensitive to how partition leadership is distributed. When leaders concentrate on a subset of brokers, those nodes saturate while the rest of the cluster is idle, and tail latency rises well before average latency shows anything. AceMQ diagnoses and rebalances.
Challenge
Latency spikes appear after a rolling restart, a node replacement, or organic partition growth. Leadership ends up unevenly distributed, so a few brokers carry disproportionate write load. Underneath, the storage class may not deliver the sustained write throughput the cluster assumed, Kubernetes CPU limits may be throttling the reactor threads, or a topic's partition count may be too low to spread load regardless of leadership. The symptom is the same in every case.
Environment
Redpanda clusters on Kubernetes or cloud instances with local NVMe or network-attached storage, serving latency-sensitive producers.
Approach
AceMQ correlates per-broker leadership counts, disk write latency, and CPU throttling metrics against the latency percentiles to identify which layer is actually saturated. Leadership is rebalanced and partition assignment corrected first, then storage class and resource limits are addressed where the hardware is genuinely the constraint.
Solution
- 1Per-broker Raft leadership count and partition distribution analysis correlated with produce latency percentiles
- 2Leadership rebalancing and partition reassignment to spread write load evenly across the cluster
- 3Disk write latency and sustained throughput validation against the storage class actually provisioned
- 4Kubernetes CPU limit and reactor thread review to eliminate throttling of the seastar shard threads
- 5Partition count review for topics whose parallelism is too low for their throughput regardless of leadership placement
- 6Producer batching, linger, and acknowledgment settings tuned against the durability requirement rather than defaults
Outcome
Tail latency comes back into range and stays there through rolling restarts, because leadership rebalancing is applied as an operational practice rather than a one-time fix. Where storage was the real constraint, the sizing recommendation is backed by measured throughput.
Technologies
KubernetesApache KafkaRelated Use Cases
Redpanda Production Support
Named-engineer 24/7 support for Redpanda clusters covering node recovery, consumer lag incidents, upgrades, and client-side failures.
Redpanda Tiered Storage Assessment
Designing tiered storage for long retention so historical data lives in object storage without local disk dictating how long you can keep it.
Apache Flink Streaming Readiness Assessment
Evaluating whether a proposed streaming workload belongs on Flink, and what the exactly-once, state, and operational requirements will really cost.
Redpanda Migration from Apache Kafka
Migrating from Kafka to Redpanda with client compatibility testing, ACL and schema registry translation, and a staged cutover per topic.
Apache Flink Checkpoint Timeout Remediation
Resolving checkpoint timeouts under backpressure where RocksDB state has grown past what the configured checkpoint interval can absorb.
Apache Flink State Backend and Scaling Design
Designing state backend, key partitioning, and rescaling strategy for large-state Flink jobs that must restart without hours of downtime.
Facing a Redpanda Production Issue?
AceMQ's senior Redpanda engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.