Most Kafka clusters look healthy on a dashboard right up until the moment they do not.
That is not a monitoring failure. Monitoring reports the present. A health check is about the conditions you have not hit yet — the broker that has not failed, the rebalance that has not cascaded, the traffic spike that has not arrived — and whether the cluster's design survives them.
Topic and Partition Design
The first thing to examine, because it is the hardest to change later and the most frequently wrong.
Partition counts chosen when the workload looked different. Too few and consumer parallelism is capped no matter how many instances you add. Too many and you pay in metadata overhead, rebalance duration and recovery time after a broker failure.
Key distribution. Whether the partitioning key actually spreads load, or whether one partition is carrying a disproportionate share because the key is skewed. Hot partitions look like a Kafka performance problem and are a design problem.
Topic proliferation. Estates that grew organically accumulate topics nobody owns, with settings nobody chose, retained for reasons nobody remembers.
Replication and Durability
The gap to look for here is between what the business believes about data loss and what the configuration actually guarantees.
Replication factor and min.insync.replicas together determine how many broker failures you survive with acknowledged writes intact. A replication factor of three with min.insync.replicas of one does not deliver what most people assume it does.
Producer acks settings have to match. A durable topic configuration and producers using acks=1 is a common and quiet mismatch.
Rack or availability zone awareness. Three replicas that all landed in the same failure domain are one replica wearing a disguise.
Unclean leader election. Whether it is enabled, and whether anyone made that choice deliberately.
Consumer Behaviour and Capacity
Consumer group configuration is where survivable incidents turn into outages. Session timeouts, heartbeat intervals and max poll intervals that are fine at normal load can guarantee a rebalance storm the moment processing slows. The mechanics are in why Kafka consumer groups rebalance.
Lag patterns over time, not lag right now. A consumer that recovers from lag slowly is telling you something about headroom that a current-value dashboard will not. See diagnosing consumer lag.
Retention against disk growth. Retention settings measured against actual growth rates, with a date attached to when disks fill.
Broker and controller headroom, including whether the cluster survives losing one broker at peak — not at average.
Version and upgrade posture, particularly for anyone still on ZooKeeper. See KRaft production readiness.
What Good Output Looks Like
A prioritised list, not an inventory.
Fix now: findings where a single broker failure, a routine restart or a normal traffic peak causes data loss or an outage.
Fix before the next growth step: design choices that hold today and will not hold at the next capacity tier.
Leave alone: the things that are fine, stated explicitly. This part matters — a report that flags everything gives the team no way to decide what to do first, and a hundred undifferentiated observations means the ranking work was skipped.
AceMQ runs Kafka health checks on production estates and provides ongoing Kafka support afterwards. If you want to know what will break before it does, get in touch.
FAQ
What is a Kafka health check?
A structured review of a running cluster against how it is actually used — topics and partitions, replication, consumer behaviour, capacity, monitoring — producing a prioritised list of what breaks first.
How is that different from monitoring?
Monitoring reports the present. A health check evaluates conditions you have not hit yet — broker failure, rebalance under load, a traffic spike.
What problems does it usually find?
Partition counts sized for an older workload, durability settings that do not match what the business assumes, consumer settings that guarantee rebalance storms, retention that will fill disks, and monitoring blind to consumer lag.
When should we run one?
Before a traffic increase, around a version upgrade, after a survived incident, when the original team has moved on, or periodically for critical clusters.
Does it require production access?
Visibility rather than access. Configuration exports, consumer group state and metrics history plus a session with your operators is usually enough.
What do you get at the end?
Findings ranked into fix now, fix before the next growth step, and leave alone — with the failure condition and the fix stated for each.
