Guide · Kafka

Kafka Migration Guide: ZooKeeper to KRaft Step by Step, 2026

Kafka migrations fail at the cutover, not the copy. Replicating topics between clusters is well-trodden; moving consumers across without replaying or skipping messages is where the plans come apart. This guide takes the three migrations teams actually run and sequences the decisions in each.

Tyler Eastridge

By Tyler Eastridge, Head of Operations

LinkedIn · Updated · 11 min read · 7 sections

On this page

Kafka migration in one paragraph

Kafka migration in one paragraph

To migrate Kafka from ZooKeeper to KRaft, get the brokers to 3.9.1 or later, start a KRaft controller quorum with the existing cluster ID and migration enabled, restart the brokers in migration mode and then as KRaft brokers, and finalize after a soak. You can roll back until you finalize. Section 1 is the full runbook.

There are three different things called a Kafka migration and they share almost nothing. Moving from ZooKeeper to KRaft is a metadata-layer change on the same cluster, it is a prerequisite for the 4.x line because 4.0 removed ZooKeeper entirely, and it runs through a bridge mode that requires the brokers to be on a recent enough 3.x release first. Moving between platforms, whether self-managed to MSK, Confluent to open source, or anything to Redpanda, is a data migration where MirrorMaker 2 copies topics and translates consumer offsets, and where the cutover sequence decides whether you lose messages. Moving off Kafka entirely, usually to RabbitMQ, is an architecture decision that should be made on messaging pattern rather than on operational frustration. Each has a different risk, and only the first has a deadline set by the project.

ZooKeeper to KRaft migration, step by step

To migrate Kafka from ZooKeeper to KRaft: bring the brokers to Apache Kafka 3.9.1 or later with inter.broker.protocol.version=3.9, start a KRaft controller quorum that holds the existing cluster ID and has migration turned on, restart the brokers in migration mode, restart them again as KRaft brokers, wait, and then finalize by taking the controllers out of migration mode. You can roll back at any point until you finalize, and not after. Kafka 4.0 removed ZooKeeper entirely, so this migration is a prerequisite for every 4.x release, which is why it comes before the upgrade rather than instead of it.

The runbook below follows the Apache Kafka 3.9 KRaft operations documentation, last modified on 23 October 2026 and read on 28 October 2026. Check the page for your exact patch release before a migration window, and rehearse on a copy of the cluster first.

Step 0: Know what blocks the migration

Three conditions stop or stall it. Features KRaft does not fully implement stop it. A broker running several log directories with a failed one cannot migrate until the directory is repaired (KAFKA-16431). And if the ZooKeeper Security Migration Tool was run before, the migration can stall on a malformed /migration node (KAFKA-19480), which you clear with delete /migration in zookeeper-shell.sh. Find all three on the old cluster before you book the window. Do not change the metadata version (inter.broker.protocol.version) while a migration is in progress.

Step 1: Get every broker to the bridge release

Apache requires brokers on 3.9.1 with inter.broker.protocol.version=3.9. Version 3.9.2, released on 21 February 2026, is the last 3.x release and adds a compatibility flag for a known difference in what an AlterConfigPolicy receives between ZooKeeper and KRaft modes (KIP-1252), so use it if you run such a policy. If you use ACLs, migrate on 3.6.2 or later: CVE-2024-27309 could make Kafka ignore DENY rules during a migration on 3.5.0 to 3.6.1 (see the Kafka CVE register). Turn on TRACE logging for the migration on each controller with log4j.logger.org.apache.kafka.metadata.migration=TRACE.

Step 2: Read the cluster ID and provision the controllers

bin/zookeeper-shell.sh localhost:2181 get /cluster/id

The KRaft controllers must carry that cluster ID. Give each controller a node.id that does not collide with any broker.id, because brokers and controllers share one ID space. Use three or five controllers: a majority must be alive, so three tolerate one failure and five tolerate two. A controller that is ready to migrate looks like this (Apache's sample):

process.roles=controller
node.id=3000
controller.quorum.bootstrap.servers=localhost:9093
controller.listener.names=CONTROLLER
listeners=CONTROLLER://:9093
# Enable the migration
zookeeper.metadata.migration.enable=true
# ZooKeeper client configuration
zookeeper.connect=localhost:2181
# Brokers' inter-broker listener, so the controller can send RPCs to them
inter.broker.listener.name=PLAINTEXT
bin/kafka-storage.sh format --standalone --cluster-id=<zk-cluster-id> -c config/kraft/controller.properties
bin/kafka-server-start.sh config/kraft/controller.properties

Apache's example starts one standalone controller. For a three or five node quorum, provision the others as the KRaft documentation describes for your Kafka release, all with the same cluster ID and the migration flag on.

Step 3: Restart the brokers in migration mode (the hybrid phase)

Add the controller quorum settings and the migration flag to each broker, keep broker.id set to a non-negative integer, and restart the brokers one at a time. Brokers still run in ZooKeeper mode, with the KRaft controller sending them controller RPCs.

broker.id=0
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
inter.broker.protocol.version=3.9
# Enable the migration
zookeeper.metadata.migration.enable=true
zookeeper.connect=localhost:2181
controller.quorum.bootstrap.servers=localhost:9093
controller.listener.names=CONTROLLER

The migration starts by itself once the last broker has restarted. The active controller logs Completed migration of metadata from Zookeeper to KRaft when the metadata copy is done.

Step 4: Convert the brokers to KRaft (the dual-write phase)

Reconfigure each broker as a KRaft broker and restart it, one at a time: replace broker.id with node.id and keep the same number, add process.roles=broker, remove the ZooKeeper settings and zookeeper.metadata.migration.enable, remove control.plane.listener.name if you set it, and drop inter.broker.protocol.version, because KRaft uses the metadata.version feature instead. If the broker uses kafka.security.authorizer.AclAuthorizer, change authorizer.class.name to org.apache.kafka.metadata.authorizer.StandardAuthorizer at this restart. The controllers keep writing metadata to ZooKeeper while this goes on.

process.roles=broker
node.id=0
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
controller.quorum.bootstrap.servers=localhost:9093
controller.listener.names=CONTROLLER

Step 5: Soak, then finalize

Apache notes that some users wait a week or two before finalizing, which means keeping ZooKeeper running that long, to validate KRaft mode on the real workload. During the soak, check bin/kafka-metadata-quorum.sh --bootstrap-server host:9092 describe --status, confirm under-replicated and offline partitions are 0, and run the failure drills you would run on any new cluster. To finalize, remove zookeeper.metadata.migration.enable and the ZooKeeper client settings from each controller and restart the controllers one at a time. After that, ZooKeeper can be deprovisioned, and the migration cannot be reverted.

Step 6: Then upgrade to Kafka 4.x

Brokers can upgrade to 4.0 and later from any KRaft cluster with software and metadata versions of at least 3.3.x. Before the upgrade window, move broker hosts to Java 17 (Apache supports 17, 21 and 25 for brokers and recommends the latest LTS), and fix every script, admin call and config that still points at a ZooKeeper connection string, such as --zookeeper or zookeeper.connect, because 4.x has no ZooKeeper mode: use --bootstrap-server. On 28 October 2026 Apache maintains 4.4, 4.3 and 4.2.

Rolling back

Revert steps depend on how far you got, and every one needs the ZooKeeper controller epoch checked first: Apache added that check to its documentation on 23 October 2026 (KAFKA-20488), because a ZooKeeper controller elected with a lower epoch than the KRaft one cannot change partition state.

Last step completedHow to revert
Controllers provisioned, brokers not yet in migration mode (steps 1 and 2)Deprovision the KRaft controller quorum. Nothing else changed.
Brokers in migration mode (step 3)Deprovision the quorum. In zookeeper-shell.sh, compare get /controller_epoch with the kraft_controller_epoch field in get /migration, and if the KRaft epoch is higher, set /controller_epoch above it. Run delete /controller, then get /migration and delete /migration. On each broker remove the migration, controller.listener.names and controller.quorum.bootstrap.servers settings, make sure each broker uses broker.id rather than node.id, and roll the brokers.
Brokers converted to KRaft (step 4)Two rolls. First, remove process.roles, restore broker.id and zookeeper.connect, keep zookeeper.metadata.migration.enable=true, and roll. Then deprovision the quorum, do the epoch check, delete /controller and /migration, remove the migration settings and roll a second time.
Finalized (step 5)You cannot revert.

Sources: Apache Kafka KRaft operations (3.9), the 4.4 upgrade notes, the 4.0 upgrade notes, the Java version page and the Apache Kafka CVE list.

Not sure which blockers your cluster has? A Kafka health check finds them before you book the window. If you cannot migrate yet and need 3.x brokers and ZooKeeper to stay patched meanwhile, that is OSSeva's Kafka and ZooKeeper extended support, a separate service from AceMQ.

Choosing the destination platform

Self-managed Apache Kafka, Amazon MSK, Confluent Platform, Confluent Cloud and Redpanda differ less in the protocol than in what you are responsible for and what a support contract can reach. The decision that matters operationally is which layer you keep: MSK gives you the brokers and leaves topics, consumers and connectors with you, while a fully managed service takes more and constrains more. Price the destination against the whole operating cost, engineer time included, rather than against the infrastructure line alone.

Copying the data: MirrorMaker 2 and offset translation

MirrorMaker 2 replicates topics between clusters and, importantly, translates consumer group offsets, because an offset on the source cluster does not mean the same thing on the target. It runs as a Connect cluster and needs its own capacity planning; a MirrorMaker deployment sized for the steady state will fall behind during the initial backfill. Decide early whether topic names are remapped with a cluster prefix or preserved, because that choice reaches into every client configuration and is painful to reverse mid-migration.

Response targets from 15 minutes, by plan. Named senior engineers.

Contact Us

The cutover: where migrations actually fail

The safe sequence is to replicate until the target is caught up, stop producers, let consumers drain the source to zero lag, verify offsets translated correctly, then repoint producers and consumers together. Teams get into trouble by moving consumers before producers, which replays or skips depending on which way the offsets translated, or by running dual-write without a plan for reconciling the two clusters afterwards. Agree the rollback trigger before the window opens, and confirm the source cluster stays up and untouched long enough to roll back to.

If the destination is Kubernetes

Running Kafka on Kubernetes changes the failure modes rather than removing them: storage class and volume behaviour decide what a broker restart costs, pod anti-affinity decides whether a node failure takes a whole replica set, and rolling updates need to respect partition leadership rather than just pod readiness. It is a viable destination and a poor place to learn Kafka operations for the first time during a migration.

Migrating off Kafka, and when that is right

Some estates run Kafka for work that is a queue rather than a log: task distribution, request-reply, per-message routing, selective consumption. Those are RabbitMQ patterns, and forcing them onto Kafka produces the partition and consumer-group contortions that make teams describe Kafka as complicated. Make the call on messaging pattern, not on operational frustration, because moving to a broker you also do not operate well solves nothing.

From our engagements. American National Insurance needed change data capture for SQL Server and evaluated Kafka for it. Kafka was judged too complex to run for the job, so AceMQ designed standalone Debezium publishing directly to RabbitMQ. That avoided running a Kafka cluster and Kafka Connect, whose operations would have been out of proportion to the data volume, and delivered full change capture with more flexible routing (the case).

Prove recovery on the new cluster before you trust it

A migrated cluster inherits none of the confidence the old one earned. Before the source is decommissioned, run the recovery you would need in an incident on the target: restore or replay to a known point, confirm the retention settings match what downstream systems assume they can replay, and write down the resulting RPO. Migrations are the most common moment for a disaster recovery plan to quietly stop being true.

Frequently asked questions

Do I have to migrate from ZooKeeper to KRaft?

Yes, if you intend to run Kafka 4.0 or later. The 4.0 release removed ZooKeeper entirely rather than deprecating it, so KRaft is a prerequisite for every subsequent upgrade. The migration runs through a bridge mode that requires the brokers to be on a recent enough 3.x release first.

What should I check before migrating Kafka from ZooKeeper to KRaft?

Be on Kafka 3.9.1, the last bridge release, with inter.broker.protocol.version set to 3.9. Check for features KRaft does not fully support and for failed log directories, which block migration. Provision three or more controllers with the existing cluster ID and node IDs that do not collide with broker IDs, and decide the rollback rule up front: you can revert to ZooKeeper only until you finalize. Then audit tooling that still uses --zookeeper before moving to 4.x.

What Java version does Kafka 4.0 need?

Java 17 for brokers, Kafka Connect and the tools, and Java 11 or later for clients and Kafka Streams applications, per the Apache Kafka 4.0 upgrade notes. Plan the JDK change on the broker hosts as part of any Kafka upgrade to 4.0, alongside the KRaft migration that has to come first.

How are consumer offsets handled in a Kafka cluster migration?

MirrorMaker 2 translates consumer group offsets between clusters, because the same offset number does not refer to the same message on the target. Verifying that translation before repointing consumers is the step that separates a clean cutover from one that replays or skips messages.

What is the safe cutover sequence?

Replicate until the target is caught up, stop producers, let consumers drain the source to zero lag, verify translated offsets, then move producers and consumers together. Moving consumers before producers is the most common way to lose or duplicate messages.

Should we migrate off Kafka to RabbitMQ?

Only if the workload is queue-shaped rather than log-shaped: task distribution, request-reply, per-message routing and selective consumption are RabbitMQ patterns. Make the decision on messaging pattern rather than on operational difficulty, which usually follows the team rather than the broker.

How do I migrate from ZooKeeper to KRaft step by step?

Get every broker to Apache Kafka 3.9.1 or later with inter.broker.protocol.version=3.9. Read the cluster ID from ZooKeeper and start KRaft controllers carrying it, with zookeeper.metadata.migration.enable=true. Restart the brokers in migration mode, then restart each one as a KRaft broker with process.roles=broker and a node.id equal to its old broker.id. After a soak period, remove the migration flag from the controllers and restart them one at a time to finalize. Then upgrade to 4.x.

Can I roll back a ZooKeeper to KRaft migration?

Yes, until you finalize. The steps depend on how far you got, and each needs a check of the ZooKeeper controller epoch against the KRaft epoch so that the ZooKeeper controller that takes over starts with a higher epoch. After finalizing, reverting to ZooKeeper is not possible, which is why Apache notes that some users wait a week or two before finalizing.

Does the ZooKeeper to KRaft migration need downtime?

Apache's procedure restarts brokers in a rolling fashion to avoid affecting cluster availability, and the migration itself starts automatically once the last broker has restarted in migration mode. Rehearse on a copy of the cluster, because clients and tooling that still assume ZooKeeper are the usual source of surprises.

Free Consultation

Get Expert Eyes on Your Kafka Cluster

Whether you're troubleshooting a production incident, planning a migration, or want a second opinion on your architecture — our team is ready. No pitch, just answers.

Email Us