A Kafka rebalance is the process of redistributing partitions among consumers in a group. It is triggered whenever membership changes — a consumer joins, leaves, or is declared dead — or when the number of partitions on a subscribed topic changes. A healthy group rebalances occasionally and briefly. A group rebalancing every few minutes has a configuration problem, and the resulting pauses are usually mistaken for a broker or throughput issue.
Kafka rebalancing is not a fault in itself. It is how Apache Kafka distributes partitions to consumers and keeps work evenly spread as a deployment scales. Understanding how Kafka rebalancing works matters because rebalancing issues arise when it fires far more often than the workload warrants.
The mechanism is worth understanding, because almost every fix is a consumer configuration change rather than anything done to the cluster.
What Triggers a Kafka Rebalance?
Four things, and only the first two are intentional.
A new consumer joins the group. Scaling up, or a pod starting after a deployment. Expected — when a new consumer joins the group, the coordinator spreads the new partitions among the consumers already running.
A consumer leaves cleanly. A graceful shutdown that sends a leave request. Also expected. Whenever a consumer joins or leaves, redistribution follows.
A consumer is declared dead. This is where the trouble lives. The group coordinator initiates a rebalance the moment it decides a consumer is gone, which happens when either of two clocks runs out:
session.timeout.ms(default 45 seconds). The consumer sends heartbeats on a background thread everyheartbeat.interval.ms(default 3 seconds). Miss enough heartbeats and the coordinator evicts you. This catches genuine crashes, network partitions and long GC pauses.max.poll.interval.ms(default 5 minutes). The gap between successivepoll()calls on the main thread. Heartbeats keep flowing on the background thread, so the consumer looks alive — but if the application takes too long processing a batch, the coordinator evicts it anyway.
That second one causes most unexplained rebalances in production. The consumer has not crashed. It is just processing slowly, and Kafka cannot distinguish a slow consumer from a stuck one.
The number of partitions changes. When new partitions are added to a Kafka topic, reassignment follows. Subscription changes do the same.
The last two are rebalance events you would rather not have. Triggering a Kafka rebalance costs the same whatever the cause, and unnecessary rebalance activity is what turns a working consumer group into a stalled one.
What Actually Happens During a Kafka Rebalance?
All consumers in the group stop consuming, must rejoin the group, and wait before they can resume. The group leader — one of the consumers, not a Kafka broker — sends a request to the group coordinator, computes how partitions are assigned, and distributes the result. Partitions are revoked first, then handed back out — and which consumer ends up with specific partitions can change on every round.
The important property is that this is a synchronization barrier: all consumers in a consumer group move together, and the slowest member sets the pace. One consumer taking thirty seconds to rejoin means the whole group waits thirty seconds. No message processing happens during the rebalancing process, so a backlog builds and lag climbs even though every broker in the Kafka cluster is healthy.
This is why rebalances and lag are so often confused. If you are chasing growing lag rather than the rebalances causing it, diagnosing Kafka consumer lag covers the symptom side.
Eager versus cooperative rebalancing
The Kafka rebalance protocol has two generations, and which one you are running changes the cost of every rebalance enormously.
Eager rebalancing is the original, stop-the-world behaviour. Every consumer revokes *all* of its assigned partitions before the new assignment is computed. Even if the reassignment only moves two partitions out of two hundred, everything stops. The pause scales with group size and rejoin time, not with how much actually changed.
Incremental rebalancing revokes only the partitions that are actually moving. Consumers keep processing everything they retain. The cooperative rebalancing protocol runs as two lighter rounds instead of one heavy one, so the impact of rebalancing scales with the size of the change rather than with how many members there are. This incremental cooperative approach is what most estates should be running.
This has been available since Kafka 2.4 and is not always the default in an older Kafka client. Static membership arrived slightly earlier, in Kafka 2.3. Switching is one line:
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor⚠️ Do not just set this and deploy. Moving from eager to cooperative requires a two-phase rolling upgrade — first deploy with both strategies listed so the group can agree, then remove the eager one on a second pass. Changing it in one step on a running group will fail the rebalance.
Newer releases go further again. The new consumer rebalance protocol moves assignment computation to the broker and removes the global synchronization barrier entirely, so consumers no longer all stop together. This new protocol reached general availability in Apache Kafka 4.0. If you are on a recent release it is worth evaluating — but most production estates are not there yet, and the sticky assignor delivers the majority of the benefit today.
Partition assignment strategies
| Assignor | Behaviour | Rebalance cost |
|---|---|---|
RangeAssignor | Per-topic contiguous ranges | Eager; can distribute unevenly across multiple topics |
RoundRobinAssignor | Even spread across all partitions | Eager; reassigns everything |
StickyAssignor | Even spread, preserves prior assignment where possible | Eager, but moves fewer partitions |
CooperativeStickyAssignor | Sticky, and revokes only what moves | Incremental — the one to want |
Stickiness matters beyond the pause itself. When a consumer keeps the same topic partitions across a rebalance, it keeps warm local state — caches, in-flight aggregations, Kafka Streams state stores. Reshuffling needlessly throws that away, and the recovery cost often exceeds the rebalance itself.
If you use Spring Kafka or another framework wrapper, this is set through the same underlying property — the framework does not change which strategies are available to you.
The Kafka Rebalance Storm: Why It Never Recovers
The rebalancing problems that bring people to us almost always follow one shape:
- Processing slows down — a downstream API degrades, or a batch gets larger.
- A consumer exceeds
max.poll.interval.msand is evicted. - The group rebalances. Everyone stops.
- The backlog grows during the pause.
- On resume, each consumer pulls a full batch against a bigger backlog and takes *longer* to process it.
- Another consumer exceeds the interval. Return to step 3.
Data processing throughput collapses while the cluster itself stays healthy.
It never recovers on its own, because each new Kafka rebalance makes the next one more likely — which is exactly why teams start investigating the cluster rather than how they configure the consumer.
Breaking the loop is usually a matter of making batches finish inside the interval:
- Reduce
max.poll.records. The most effective single change. Fewer records per poll means less time between polls. Start by halving it. - Raise
max.poll.interval.msto genuinely exceed worst-case processing time, not typical time. Measure the p99, do not estimate. - Move slow work off the polling thread. If processing involves a network call of unpredictable duration, the poll loop is the wrong place for it.
- Do not increase
session.timeout.msto fix this. It is a different clock and will not help — it only delays detection of genuinely dead consumers.
Static membership
Where consumers restart routinely — rolling deployments on Kubernetes, most obviously — this stops the restart from triggering one at all.
Assign each consumer a stable group.instance.id. On restart inside the timeout window, the coordinator recognises the returning member and hands back its previous partitions without redistributing anything:
group.instance.id=consumer-3
session.timeout.ms=45000The trade-off is honest: a genuinely dead member is not replaced until its session times out, so its partitions sit unconsumed for that window. Decide whether rebalancing is necessary on restart before enabling it. Set the timeout to comfortably exceed your rolling restart time and no more.
There is also a broker-side setting worth knowing for startup. group.initial.rebalance.delay.ms (default 3 seconds) makes the coordinator wait briefly before the first rebalance, so consumers starting together form one group rather than rebalancing repeatedly as each arrives. It is a small thing that removes a lot of noise from Kubernetes deployments.
What to monitor
Watch three things, and watch them together:
- Rebalance rate. The signal itself. Anything above occasional deserves investigation — and a consumer stuck in a rebalance loop shows up here first.
- Rebalance duration. Rising duration means consumers are slow to rejoin — often the same slow processing that triggered the rebalance.
- Consumer lag during rebalances. Confirms whether pauses are actually costing you throughput or are short enough not to matter.
A consumer group that runs a Kafka rebalance a few times a day around deployments is fine. One rebalancing every few minutes is telling you max.poll.interval.ms is being exceeded, and no amount of cluster tuning will change that.
Seeing rebalance storms or unexplained consumer pauses? These almost always sit in consumer
configuration rather than the cluster, which is why broker metrics look clean while throughput
collapses. AceMQ runs structured Kafka assessments that find them —
FAQ
What triggers a Kafka consumer group rebalance?
A consumer joining, leaving cleanly, or being declared dead by the group coordinator — either through missed heartbeats past session.timeout.ms or through exceeding max.poll.interval.ms between polls. A change in the partition count of a subscribed topic also triggers one.
Why does rebalancing cause latency and backlog?
Because it is a synchronization barrier. Consumers stop consuming while the group rejoins and waits for a new assignment, so the slowest member sets the pause for everyone. Nothing is processed during that window and lag accumulates.
What is cooperative rebalancing in Kafka?
An incremental protocol that revokes only the partitions actually changing hands, letting consumers keep processing everything else. Eager rebalancing revokes every partition from every consumer regardless of how little is moving.
How do I stop frequent Kafka rebalances?
Find out which clock is expiring. If batches take too long, lower max.poll.records and raise max.poll.interval.ms past the worst case. If restarts are the cause, use static membership. Then move to the cooperative sticky assignor to make the remaining rebalances cheaper.
What is static group membership?
A stable group.instance.id per consumer that lets the coordinator recognise a restarting member and return its previous partitions without a rebalance, provided it returns inside the session timeout.
Which partition assignment strategy should I use?
CooperativeStickyAssignor for most workloads — it minimises both partition movement and rebalance disruption. Migrate to it over two rolling deployments rather than changing the setting in one step.
Does adding partitions trigger a Kafka rebalance?
Yes. Increasing the partition count on a subscribed topic causes the group to reassign, so treat partition changes as a scheduled operation rather than a routine one.
Is a Kafka rebalance the same as reassigning partitions across brokers?
No, and the shared vocabulary causes real confusion. A consumer group rebalance redistributes partitions among *consumers*. Moving partition replicas between *brokers* is partition reassignment — a separate cluster operation with different tooling and different risks.
