The Kafka Migration Guide

Kafka 4.0 runs on KRaft only. ZooKeeper is not deprecated in 4.0, it is gone, which makes this migration a prerequisite for every subsequent upgrade rather than an optional modernisation. The path runs through a bridge mode where a KRaft controller quorum takes over metadata while the existing brokers keep serving, so the brokers must already be on a 3.x release new enough to support migration before you start. Plan controller quorum sizing separately from broker sizing, and audit tooling for anything that still takes a ZooKeeper connection string, because those break silently.

Self-managed Apache Kafka, Amazon MSK, Confluent Platform, Confluent Cloud and Redpanda differ less in the protocol than in what you are responsible for and what a support contract can reach. The decision that matters operationally is which layer you keep: MSK gives you the brokers and leaves topics, consumers and connectors with you, while a fully managed service takes more and constrains more. Price the destination against the whole operating cost, engineer time included, rather than against the infrastructure line alone.

MirrorMaker 2 replicates topics between clusters and, importantly, translates consumer group offsets, because an offset on the source cluster does not mean the same thing on the target. It runs as a Connect cluster and needs its own capacity planning; a MirrorMaker deployment sized for the steady state will fall behind during the initial backfill. Decide early whether topic names are remapped with a cluster prefix or preserved, because that choice reaches into every client configuration and is painful to reverse mid-migration.

The safe sequence is to replicate until the target is caught up, stop producers, let consumers drain the source to zero lag, verify offsets translated correctly, then repoint producers and consumers together. Teams get into trouble by moving consumers before producers, which replays or skips depending on which way the offsets translated, or by running dual-write without a plan for reconciling the two clusters afterwards. Agree the rollback trigger before the window opens, and confirm the source cluster stays up and untouched long enough to roll back to.

Running Kafka on Kubernetes changes the failure modes rather than removing them: storage class and volume behaviour decide what a broker restart costs, pod anti-affinity decides whether a node failure takes a whole replica set, and rolling updates need to respect partition leadership rather than just pod readiness. It is a viable destination and a poor place to learn Kafka operations for the first time during a migration.

Some estates run Kafka for work that is a queue rather than a log: task distribution, request-reply, per-message routing, selective consumption. Those are RabbitMQ patterns, and forcing them onto Kafka produces the partition and consumer-group contortions that make teams describe Kafka as complicated. Make the call on messaging pattern, not on operational frustration, because moving to a broker you also do not operate well solves nothing.

A migrated cluster inherits none of the confidence the old one earned. Before the source is decommissioned, run the recovery you would need in an incident on the target: restore or replay to a known point, confirm the retention settings match what downstream systems assume they can replay, and write down the resulting RPO. Migrations are the most common moment for a disaster recovery plan to quietly stop being true.