Is Kafka 4.0 (KRaft, No ZooKeeper) Production-Ready?

Kafka 4.0 represents the most significant architectural shift in the project's history: ZooKeeper, the external coordination service Kafka relied on since its earliest versions, is gone. Cluster metadata and controller consensus are now handled internally via KRaft (Kafka Raft), Kafka's own Raft-based consensus protocol. This is a genuine simplification of the operational model — one fewer distributed system to run and tune — but "newly released major architecture" and "battle-tested for your specific workload" are different claims.

This post covers what's actually changed operationally, based on a real production case that hit a significant performance issue on Kafka 4.0.0 specifically.

KRaft mode is the standard, production architecture in Kafka 4.0 — it's not a beta or preview feature at this point. Organizations are actively deploying it in production, including for demanding use cases: one organization specifically adopted Kafka 4.0.0 for a banking-sector transaction processing platform explicitly because it removes the ZooKeeper dependency, treating that architectural simplification as a deliberate, forward-looking technology choice rather than a risk to avoid.

That said, "the architecture is production-grade" and "every feature behaves identically to how it did under the ZooKeeper-based model" are different claims — and the real case referenced throughout this post is a direct illustration of exactly that gap.

A significant, specific performance regression tied to Kafka transactions. An organization running a transaction-processing platform on Kafka 4.0.0, using Spring Boot 3.2 and Java 17, had previously measured throughput around 320,000 transactions per hour using standard Kafka messaging. After adding recovery capability to their system — implemented using Kafka's transactional messaging feature — throughput dropped to 40,000-50,000 transactions per hour, and critically, scaling up the number of service instances did not improve it. The system was stuck at the same throughput ceiling regardless of horizontal scaling, which is a strong signal that the bottleneck was structural (likely coordinator-related) rather than simple resource contention.

When the team tested running the same workload without Kafka transactions, performance returned to normal — isolating the regression specifically to the transactional messaging path, not to KRaft mode, the broker configuration, or the application code more broadly.

Not necessarily broken — but it does mean transactions specifically warrant deeper validation before you commit to them for a performance-sensitive workload, since the transaction coordinator's behavior under KRaft is architecturally different from how it worked under ZooKeeper-based clusters, and that difference can manifest as exactly this kind of throughput ceiling under certain usage patterns.

What we know from this case:

• The throughput ceiling was reproducible and consistent (not intermittent)
• It didn't respond to horizontal scaling, ruling out simple resource starvation as the cause
• Removing transactions from the same workload restored normal throughput

What this points to diagnostically: transaction coordinator behavior — how transactional state is tracked, committed, and coordinated across the cluster — is a strong candidate for the bottleneck, since that's the component most directly tied to the "transactions on vs. off" behavior difference observed. A structured diagnostic engagement for this kind of issue would examine transaction coordinator placement, transaction timeout configuration, and transactional.id cardinality and partitioning as the first investigation targets.

Not necessarily — but validate your specific transactional usage pattern under realistic load before committing to it for a production-critical, latency-sensitive workload, rather than assuming general Kafka transaction guidance from pre-4.0 documentation carries over unchanged.

If your use case requires Kafka transactions (exactly-once semantics across produce/consume boundaries, or atomic multi-partition writes):

• Load test the transactional path specifically and separately from your non-transactional throughput baseline, using a representative transaction rate and transactional.id pattern
• Test whether horizontal scaling of your producer/consumer instances actually improves transactional throughput in your environment — if it doesn't, that's the same warning signal observed in the case above, worth investigating before going to production
• Review current Kafka 4.x release notes and known issues specifically for KRaft transaction coordinator behavior, since this is an actively evolving area of the codebase post-ZooKeeper-removal

If your use case doesn't strictly require transactions, evaluate whether your recovery/idempotency requirements can be met through alternative patterns — idempotent producer configuration alone (without full transactions), application-level deduplication, or a different recovery architecture — that may avoid the specific bottleneck observed here entirely.

The real case referenced throughout this post used JMeter to generate controlled, steady load at a specific transaction-per-second/minute rate — a deliberate choice specifically because it gives clear, controllable visibility into how the system responds to load ramps, rather than relying on production traffic patterns that are harder to isolate and reproduce for diagnostic purposes.

A sound validation approach:

1. Establish a non-transactional throughput baseline first, so you have a clean comparison point
2. Introduce transactions and measure the throughput delta under the same load profile
3. Test horizontal scaling explicitly — add producer/consumer instances and confirm throughput actually increases; if it plateaus, that's your signal to investigate coordinator-level bottlenecks before going further
4. Use a load-generation tool that lets you hold a steady, known transaction rate (JMeter, or Kafka's own kafka-producer-perf-test.sh for producer-side validation) rather than relying solely on organic traffic during a soft launch

If you're currently on a pre-KRaft version and planning your upgrade path, Kafka 4.0 and later are ZooKeeper-free by design — ZooKeeper support was removed, not just deprecated, as of Kafka 4.0. This means your upgrade planning needs to include a KRaft migration as part of the path, not as an optional follow-on step.

For workloads that don't use Kafka transactions heavily, this migration is generally lower-risk than the case described in this post, since the specific bottleneck identified here was isolated to transactional throughput specifically, not general KRaft cluster operation. For transaction-heavy workloads, budget explicit load-testing time for the transactional path as part of your migration plan, using the validation approach outlined above, rather than assuming pre-KRaft transaction performance characteristics carry over unchanged.

The core KRaft architecture is production-ready and is being actively adopted for demanding workloads, including financial transaction processing. Kafka transactions specifically warrant dedicated load validation before you commit to them for a performance-sensitive production workload, based directly on a real, reproducible throughput regression observed in production testing. This isn't a reason to avoid Kafka 4.0 — it's a reason to test your specific usage pattern deliberately rather than assuming full behavioral parity with pre-KRaft Kafka versions across every feature, transactions included.

A KRaft cluster keeps its metadata in a Raft quorum of controller nodes instead of in ZooKeeper. The quorum elects one active controller, which owns the metadata log: topic and partition definitions, replica assignments, leader elections, configs, ACLs. Every broker follows that log and applies it locally. Each node declares what it is through process.roles. A node with process.roles=controller only votes and serves metadata. A node with process.roles=broker only serves client traffic and partition data. A node with process.roles=broker,controller does both on the same JVM, which is what upstream calls combined mode.

Upstream documentation is clear on which of these belongs in production. Isolated controllers on their own servers are the recommended layout for production workloads. Combined mode is described as suitable for development, testing and small clusters where resource isolation is not a concern. It is supported and it works, but it is not the layout upstream recommends for critical deployments, and our experience matches that.

The reason is failure coupling. In combined mode a broker under I/O or garbage collection pressure is also a controller, so a bad broker does not just lose leadership on its partitions, it can stall quorum votes and metadata commits for the whole cluster. The metadata log has to fsync on every commit, and in combined mode that fsync competes with partition log writes on the same disks and the same page cache, so a burst of producer traffic can slow the component that coordinates recovery from it. A rolling broker restart is also a controller restart, so every routine broker upgrade or config change also exercises a controller election. With a handful of brokers and light traffic none of this matters much. With thousands of partitions and hot disks, ordinary broker incidents become cluster-wide metadata incidents. Isolating the roles is what keeps a broker problem a broker problem.

Sizing isolated controllers is easy because they hold metadata only, not partition data. Three controllers is right for most clusters and tolerates the loss of one. Five is worth it for very large estates or multi-zone deployments where you want to survive a zone loss plus a maintenance window on another node, at the cost of a slightly slower quorum. Small machines are fine: a few cores, a few GB of heap, and a modest but low-latency disk for the metadata log. Place one controller per availability zone so a zone failure cannot take the quorum with it. On Kubernetes this means a dedicated controller node pool separate from the broker pool, which we cover in Kafka on Kubernetes. Quorum placement is also part of any serious Kafka disaster recovery design, because a cluster that has lost its controller quorum cannot elect partition leaders even if every broker is healthy.

Migrating an existing ZooKeeper cluster follows a fixed order. Bring up the new controller quorum with zookeeper.metadata.migration.enable=true and the ZooKeeper connection string, so the controllers can read the existing metadata. Then restart brokers one at a time with the migration flag and the controller quorum voters configured, which moves the cluster into dual-write mode where metadata is written to both ZooKeeper and the KRaft log. Once every broker has been rolled and the metadata is verified, finalise by removing the migration flag and the ZooKeeper config from the controllers and restarting them. Up to that point you can roll back to ZooKeeper; after finalisation you cannot. Kafka 4.0 removed ZooKeeper mode entirely, so this whole procedure must run on a 3.9 bridge release. You upgrade to 3.9, migrate, finalise, and only then move to 4.x.

Our rule for critical workloads is short. Isolated controllers, three nodes, one per zone, on disks separate from anything else. Combined mode only when the cluster has fewer than a handful of brokers and the reason is written down. When we take over a cluster running combined mode at scale, splitting the roles is usually our first change, before partition strategy or capacity. If you want that review done on your estate, our Kafka consulting team can scope it, and Kafka support covers the cluster afterwards with 24/7 coverage and a 15-minute P1 response.

Migrating to Kafka 4.0, or seeing an unexplained throughput ceiling with Kafka transactions in your KRaft-based cluster? Contact AceMQ for a diagnostic engagement. Not sure where to start? Talk to an AceMQ engineer.

• Kafka Support
• What Actually Causes a Kafka Producer Timeout
• Kafka Consumer Lag: Causes, Monitoring, and How to Reduce It
• AceMQ Home