On this page
How to use the checklist
Run all 30 checks against the live cluster, not the design document, and treat any failure in group 2 (durability) or group 7 (monitoring and drills) as a go-live blocker. The other groups can ship with a dated ticket. Most failures trace back to defaults: a topic created by a typo gets one partition and one replica, because auto.create.topics.enable defaults to true while default.replication.factor and min.insync.replicas both default to 1.
A production checklist is only as good as its evidence, so record the output of each command next to the check, for the audit trail and for the next incident. The commands assume the bin scripts from the Apache Kafka distribution and a --bootstrap-server address. If you would rather have an engineer run these checks on your estate and rank what they find, that is the Kafka health check.
Cluster and metadata (checks 1 to 5)
A production cluster runs a release Apache still maintains, with three or five controller-only nodes, Kafka brokers spread across failure domains, and room to lose one broker at peak load without losing availability.
| Check | Pass when | What it prevents |
|---|---|---|
| 1. Supported release | Brokers run Apache Kafka 4.4, 4.3 or 4.2, the three releases the project maintains on 28 October 2026 (4.4.0 shipped on 22 October). A ZooKeeper cluster has a dated plan to migrate, because only KRaft clusters can reach any 4.x release. | Running on a line that no longer gets bug or security fixes. The project fixes only its last three releases. |
| 2. Dedicated controllers | 3 or 5 nodes with process.roles=controller only. Three tolerate 1 controller failure, five tolerate 2. bin/kafka-metadata-quorum.sh --bootstrap-server host:9092 describe --status shows a leader and caught-up voters. | A metadata outage when one node fails. Combined broker-and-controller nodes cannot be rolled or scaled separately, and Apache advises against combined mode in critical deployments. |
| 3. Java version | Brokers, Kafka Connect and the tools run Java 17 or later. Apache supports 17, 21 and 25 for brokers and recommends the latest LTS and the latest patch release. | A failed start on upgrade, or a JVM with known disclosed vulnerabilities. |
| 4. Zone spread | broker.rack is set on every broker (the default is null). bin/kafka-topics.sh --bootstrap-server host:9092 --describe shows each partition's replicas on separate racks or zones. | Three replicas that share one failure domain, which behave like one replica. |
| 5. Headroom for one lost broker | With one broker down at peak, NetworkProcessorAvgIdlePercent and RequestHandlerAvgIdlePercent stay above 0.3, the ideal Apache's monitoring documentation gives, and the surviving disks have room to re-replicate. | A single failure that turns into overload and then a second failure. |
Checks 1 and 2 are the ones a config change cannot fix later. Moving off ZooKeeper is a migration, laid out step by step in the ZooKeeper to KRaft runbook.
Durability and replication (checks 6 to 11)
Acknowledged writes survive a broker loss only when replication factor, min.insync.replicas, acks and leader election agree with each other. Kafka's defaults for the first two are both 1, so every one of these checks is worth running on a cluster that passed its load test.
| Check | Pass when | What it prevents |
|---|---|---|
| 6. Replication factor of 3 | Every production topic shows ReplicationFactor: 3 in kafka-topics.sh --describe. default.replication.factor is set to 3 on the brokers, because its default is 1 and it applies to auto-created topics. | Losing a broker takes a partition offline, or loses data, on any topic with one replica. |
| 7. min.insync.replicas of 2 | min.insync.replicas=2 on the brokers and on each production topic (default 1). Apache's own example is replication factor 3, min.insync.replicas=2 and acks=all. | A write acknowledged by the leader alone, then lost when that leader fails. |
| 8. Producers use acks=all | No client config sets acks to 1 or 0. The default in 4.4 is all, so search client repositories and config maps for overrides. | Acknowledged records that never reached a follower. |
| 9. Idempotence stays on | enable.idempotence is true (the default). It requires acks=all and max.in.flight.requests.per.connection of 5 or less; a conflicting setting disables it unless idempotence was set explicitly. | Duplicate and reordered records after a retry. |
| 10. Unclean leader election off | unclean.leader.election.enable=false on the brokers (the default) and no topic-level override to true. | Electing an out-of-sync replica as leader, which Apache says may result in data loss. |
| 11. Internal topics replicated | offsets.topic.replication.factor is 3 (the default), transaction.state.log.replication.factor is 3 and transaction.state.log.min.isr is 2. Check clusters first built from a one-broker test environment. | Consumer offsets or transaction state living on a single broker. |
Topics, partitions and retention (checks 12 to 16)
Topic settings are the hardest to change once traffic flows, so a production cluster creates topics deliberately, sizes partitions from the consumers it will run, and sets retention on every topic.
| Check | Pass when | What it prevents |
|---|---|---|
| 12. Auto-creation off | auto.create.topics.enable=false in production. The default is true, and an auto-created topic gets num.partitions (default 1) and default.replication.factor (default 1). | A mistyped topic name that quietly becomes a real, unreplicated topic. |
| 13. Partition count from peak consumers | Each topic's partition count covers the largest consumer group you will run at peak, with headroom. Partitions can be added but not removed, and adding them changes key-to-partition mapping. | A consumer ceiling you can only raise with a migration. |
| 14. Key skew measured | Per-partition bytes and message rates are within a small multiple of the median. One partition far above the rest means a skewed key. | A hot partition that no amount of broker hardware fixes. |
| 15. Retention set per topic | retention.ms (default 7 days) and retention.bytes (default -1, unlimited) are set from what downstream systems can replay, and the disk-full date at current growth is written down. | A disk filled by a topic nobody set retention on. |
| 16. Message size limits agree | message.max.bytes (default 1048588), replica.fetch.max.bytes (1 MiB), producer max.request.size (1 MiB) and consumer max.partition.fetch.bytes (1 MiB) are raised together, or not at all, for any topic with larger records. | Records the producer accepts and a follower or consumer cannot move. |
Hosts and operating system (checks 17 to 20)
Brokers fail at the operating system before they fail in Kafka: file descriptors, memory-map limits, shared disks and an oversized heap are the usual causes. Apache's hardware and OS guidance gives starting numbers for each. On AWS or any other cloud platform, check the instance's disk and network limits against the same numbers. On Kubernetes, vm.max_map_count is a node-level kernel setting, so set it on the node image or through the platform team rather than assuming the pod inherits it.
| Check | Pass when | What it prevents |
|---|---|---|
| 17. File descriptor limit | At least 100,000 allowed file descriptors for the broker process, Apache's stated starting point. A broker needs one per log segment plus one per connection. | A broker that cannot open segments or accept connections as partitions grow. |
| 18. vm.max_map_count | vm.max_map_count exceeds twice the broker's segment count. Many Linux systems default to about 65,535, and each log segment uses 2 map areas, so 50,000 partitions need 100,000. | A broker crash from map-area exhaustion on a partition-heavy node. |
| 19. Dedicated data disks | Kafka data directories sit on their own drives, not shared with application logs or other OS activity, formatted XFS or ext4. If log.dirs lists several directories, partitions are balanced across them. | Latency spikes from logs or other processes competing for the same disk. |
| 20. Memory and heap | RAM covers the page cache: Apache's rule of thumb is write throughput multiplied by 30 seconds of buffering. The JVM heap is modest (Apache's example is 6 GB with G1 and a 20 ms pause target), leaving the rest to the page cache. | Slow reads from cold disk, or long garbage-collection pauses that drop a broker from its group. |
Producers and consumers (checks 21 to 25)
Most Kafka incidents that start in the application are Kafka client settings that were never tested against the slowest case of a real workload: a poll interval shorter than processing time for a consumer group, timeouts that contradict each other, or a library version with a known bug.
From our engagements. In the estates AceMQ supports, the recurring Kafka incidents are rebalance storms from poll intervals shorter than processing time, consumer lag on a hot partition caused by key skew, and a disk filled by a topic nobody set retention on. Checks 14, 15 and 22 exist for those three.
| Check | Pass when | What it prevents |
|---|---|---|
| 21. Producer timeouts consistent | delivery.timeout.ms (default 120000) is at least request.timeout.ms (30000) plus linger.ms (5 since Kafka 4.0, 0 before). Apache says it should be at least that sum. | Records failing earlier than the configuration implies. |
| 22. Poll and session settings fit the work | max.poll.interval.ms (default 300000) is larger than your slowest batch, using max.poll.records (default 500) to bound it. session.timeout.ms is 45000 and heartbeat.interval.ms 3000 unless you changed them for a reason. | A consumer evicted from its group while still working, which triggers a rebalance. |
| 23. Rebalance behaviour chosen | Stateful or slow-start consumers use static membership (group.instance.id), or the group runs the new protocol with group.protocol=consumer, generally available since Kafka 4.0 and not on by default. | A full stop-the-world rebalance every time a pod restarts. |
| 24. Exactly-once boundary understood | Anything reading transactional topics sets isolation.level=read_committed (default read_uncommitted). Any side effect outside Kafka, such as a database write or an API call, is idempotent by design. | Duplicates downstream and consumers reading aborted transactions. |
| 25. Client libraries patched | Java clients, and the dependency versions they bring, are on a release that fixes CVE-2026-35554, the producer buffer race that can deliver messages to the wrong topic: 3.9.2, 4.0.2, 4.1.2, 4.2.0 or later. Other languages have their own advisories. | Silent misrouting of messages, with no error reported to the producer. |
Security (checks 26 to 28)
A production cluster encrypts every listener, authenticates every client, denies by default and keeps an audit trail; the checks below are the ones auditors and attackers both reach first.
| Check | Pass when | What it prevents |
|---|---|---|
| 26. TLS on every listener | Every listener, including the inter-broker one, uses SSL or SASL_SSL. SCRAM is used only over TLS: CVE-2024-56128 allows replay of a SCRAM exchange that an attacker can read in plaintext. | Credentials and data readable on the network, or a replayed login. |
| 27. Default deny with few super users | An authorizer is set (org.apache.kafka.metadata.authorizer.StandardAuthorizer in KRaft), allow.everyone.if.no.acl.found is left at false, and super.users lists as few user principals as possible. | A resource with no ACL that is open to everyone. |
| 28. Authentication mechanisms reviewed | If OAUTHBEARER is on, the validator checks signature, issuer and audience (CVE-2026-33557 affected 4.1.0 and 4.1.1, and Kafka 4.4 refuses to start a JWKS validator without an expected audience or issuer). The JNDI and LDAP login modules stay on the disallowed list, which is the default since 3.9.1 and 4.0.0. | Forged tokens accepted by the broker, or remote code execution through a JAAS config. |
Keep authorizer and authentication logs where your audit process can reach them, because an ACL change you cannot trace turns a small incident into a long review. All three CVEs are in the Kafka CVE register with affected and fixed versions.
Monitoring, alerting and failure drills (checks 29 and 30)
Production readiness ends with two proofs: observability that fires alerts on the signals below, and a recorded incident drill showing the cluster surviving the failures it was designed for.
Check 29: alerts exist for these signals. The thresholds are from Apache's monitoring documentation unless marked as yours to set.
| Signal | Metric or command | Healthy |
|---|---|---|
| Under-replicated partitions | kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions | 0 |
| Partitions below min ISR | kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount | 0 |
| Offline partitions | kafka.controller:type=KafkaController,name=OfflinePartitionsCount | 0 |
| Active controller | kafka.controller:type=KafkaController,name=ActiveControllerCount, summed | Exactly 1 across the cluster |
| Request handler and network idle time | RequestHandlerAvgIdlePercent, NetworkProcessorAvgIdlePercent | Ideally above 0.3 |
| Consumer lag growth | records-lag-max on clients, or kafka-consumer-groups.sh --describe --group name | Alert on sustained growth and time to catch up; the threshold is yours |
| Disk time-to-full | Data-directory usage and growth rate | A fill date that you set an alert against |
Check 30: the drills are done and written down. Kill one broker at peak and record time to recover and any client errors. Roll every broker one at a time. Replace a controller, following Apache's note that a replacement should not be formatted until a majority of controllers hold the committed data. Fail over to the disaster recovery cluster with consumers attached, and write down the recovery point you measured. Finally, restore a topic configuration from a backup of your kafka-configs.sh --describe output, to prove a rebuilt broker or cluster can be made to match.
Who runs these checks
A team that has run Kafka before can work through the 30 checks in a day or two; a team building its first platform usually cannot rank what it finds. Open-source Apache Kafka is fully supportable however it is run. AceMQ works with 160+ customers and handles 40+ Sev 1 incidents a year across the platforms it supports, with a 15-minute emergency SLA on Kafka support. The same engineers run the Kafka health check, which covers these areas and returns a prioritised fix list.
Where this checklist is not enough. It does not size a cluster, review your application design or replace load testing. Fully managed services such as Confluent Cloud and Amazon MSK take over the operational checks, 2 to 4 and 17 to 20, but not the topic and client checks, which stay with your team in both.
Frequently asked questions
What is a Kafka production readiness checklist?
A list of pass-or-fail checks run against a live Kafka cluster before it carries production traffic. This one has 30, in seven groups: cluster and metadata, durability, topics and retention, hosts and operating system, clients, security, and monitoring with failure drills.
What replication settings does production Kafka need?
Replication factor 3, min.insync.replicas 2, producers on acks=all, idempotence on, and unclean leader election off. Apache's own example is replication factor 3 with min.insync.replicas 2 and acks=all, and the broker defaults for replication factor and min.insync.replicas are both 1, so they must be set.
What should I alert on for a Kafka cluster?
Under-replicated partitions, partitions below min ISR and offline partitions, which should all be 0; exactly one active controller across the cluster; request handler and network idle percentages ideally above 0.3; sustained consumer lag growth; and disk time-to-full. Apache's monitoring documentation gives the first five.
Should Kafka controllers run on the same nodes as brokers?
Not in production. Combined mode is simpler for development, but Apache advises against it in critical deployments because controllers cannot be rolled or scaled separately from brokers. Use 3 or 5 dedicated controller nodes.
Does a managed Kafka service pass these checks for me?
Partly. A managed service operates the brokers, controllers and hosts, so checks on those are the provider's. Topic design, partition counts, retention, client settings, consumer lag and the exactly-once boundary stay with your team on every managed service.