The Kafka Monitoring and Alerting Guide

Every broker, producer and consumer metric Kafka keeps is exposed over JMX. In practice that means running the Prometheus JMX exporter as a Java agent on each broker, and scraping client metrics from the applications themselves rather than inferring them from the broker. Consumer lag is the exception worth a dedicated exporter, because the broker-side view and the consumer's own view disagree in exactly the situations you care about: a consumer that has stopped polling still has committed offsets sitting in __consumer_offsets, so the group looks alive from the outside.

UnderReplicatedPartitions above zero means at least one follower has fallen out of the in-sync replica set: the cluster still serves, but you have lost redundancy. UnderMinIsrPartitionCount is more serious, because a partition below min.insync.replicas rejects every acks=all produce request, so producers are failing while the brokers look up. OfflinePartitionsCount above zero means partitions have no leader and are unavailable for both reads and writes. And ActiveControllerCount summed across the cluster must equal one; zero means no controller, two means a split brain. Those four are the page-someone list.

RequestHandlerAvgIdlePercent and NetworkProcessorAvgIdlePercent are the brokers' headroom. Sustained values under roughly 30 per cent mean the thread pools are the bottleneck and everything downstream of them gets slower, including replication, which is how a saturation problem turns into an under-replicated-partition problem. Watch ISR shrink and expand rates alongside them: a cluster that shrinks and expands its ISR repeatedly is usually short of IO or network, not short of brokers. Disk fills are the other classic, and retention is what governs them.

Lag is the distance between a partition's log end offset and the consumer group's committed offset. The absolute figure is close to meaningless on its own, because a healthy high-throughput consumer can sit thousands of messages behind and catch up in seconds. What matters is whether lag is growing, and how long the group would take to return to zero at its current consumption rate. Alert on sustained growth over a window longer than a normal deploy, and on time-to-catch-up crossing whatever the downstream system can tolerate. A static threshold pages during every rolling restart and teaches everyone to ignore it.

A rebalance halts consumption for the whole group while partitions are reassigned, so rebalance frequency is a consumption-availability metric. Track it per group. A group that rebalances on a schedule is almost always hitting max.poll.interval.ms because processing a batch takes longer than the configured interval, or losing members to session timeouts. The fix is the trigger rather than the timeout, though cooperative sticky assignment and static group membership reduce what each rebalance costs while you find it.

Producer record error rate above zero is always worth an alert, because a producer that cannot write is losing data unless the application handles the failure well, and most do not. Request latency is the leading indicator: it rises as the acknowledgement path degrades, well before timeouts start. Buffer-available-bytes falling toward zero means the client is producing faster than it can send and will start blocking or dropping, depending on configuration. All three are client-side metrics, which is why scraping only the brokers leaves you blind to the failure mode users notice first.

Page on: offline partitions above zero, under-min-ISR partitions above zero, active controller count not equal to one, producer error rate above zero, and consumer lag growing for longer than a deploy window. Ticket, do not page: under-replicated partitions that recover within minutes, request handler idle below thirty per cent, rebalance rate above baseline, disk above seventy per cent. Everything else belongs on a dashboard someone looks at during an incident, not in a pager rotation. The test of an alert is whether the person woken can do something about it at three in the morning.