The Kafka Operations Guide

Kafka 4.0 runs on KRaft only; ZooKeeper is gone. New clusters should start on KRaft, and existing ZooKeeper clusters need a planned migration before they can take the 4.x line. KRaft is production-ready for most estates, with the usual caveats about controller quorum sizing, metadata snapshot behaviour and tooling that still assumes a ZooKeeper connection string.

Partition count sets your parallelism ceiling and cannot be reduced later, so size it to the consumer count you will run at peak with headroom, not to a throughput figure. Keys decide ordering and load distribution; a key with skewed cardinality produces a hot partition that no amount of broker hardware fixes. Re-partitioning a topic in production is a migration, so get this right before the topic carries traffic.

Lag is the distance between the last produced offset and the last committed one, per partition. A lag number tells you almost nothing; the rate of change does. Lag that grows is a consumer that cannot keep pace, a hot partition, or a rebalance in progress. Alert on sustained growth and on time-to-catch-up, not on a static threshold that pages during every deploy.

A rebalance stops consumption for the group while partitions are reassigned. They are triggered by members joining or leaving, by session or poll timeouts, and by topic changes. Most rebalance storms are a poll interval shorter than the processing time, or a consumer that dies and restarts in a loop. Cooperative sticky assignment and static membership cut the cost; fixing the trigger removes it.

A producer timeout is the broker's acknowledgement path taking longer than the client will wait: under-replicated partitions, a leader election, a full request queue, or an acks=all write waiting on a slow follower. Fix the path, not the timeout. Idempotent producers and transactions give exactly-once within Kafka; they do not make the downstream side effect exactly-once, which is where most exactly-once claims quietly fail.

Retention is a replay budget. Time-based retention sets how far back a consumer can recover; size-based retention protects the disk. Set both, per topic, from what downstream systems can actually reprocess, and remember that compacted topics keep the latest value per key rather than a window. Retention is also the disk-full incident waiting to happen on any topic where nobody set it.

Authentication with SASL (SCRAM, or OAuth where the estate has an identity provider), authorization with ACLs per principal and topic, TLS on every listener including inter-broker, and secrets kept out of client configuration files. Do this before the cluster carries regulated data, because retrofitting ACLs onto a hundred consumer groups is a project of its own.

A single cluster across availability zones survives a zone; it does not survive a region. Cross-region DR is a second cluster with MirrorMaker 2 or cluster linking, and the choice between active-passive and active-active is set by the recovery point you can accept and whether consumers can resume from mirrored offsets. Write the RPO down, then test the failover with consumers attached and time it.

Kafka on Kubernetes works with an operator, persistent volumes whose failure domain you have checked, and pod anti-affinity across zones. The provider differences are in storage classes and load balancing of long-lived connections; the failure modes are the same as any stateful workload on a scheduler: a drain during a leader election, a volume that is zonal when you assumed regional.

Every step above is routine for a team that has done it before and a project for one that has not. The alternatives to learning it during an incident are a support contract, if the platform team can hold the pager, or managed operations, if it cannot. A health check is the honest way to find out which you need.