Running RabbitMQ on Kubernetes is not running RabbitMQ on bare metal. The monitoring signals that matter on a static three-node VM cluster are not the same signals that tell you whether a RabbitMQ StatefulSet is healthy inside a Kubernetes cluster you may not fully control. And the failure modes are different — node evictions, pod rescheduling, shared-tenant I/O contention, and the specific way Raft consensus interacts with the Kubernetes networking stack can produce outages that bare-metal operators never encounter.
We've troubleshot enough of these incidents at this point that the signal-to-noise ratio of what actually matters has become clear. If you monitor only five things on a Kubernetes RabbitMQ cluster, monitor these five. They are the metrics that, in our experience across financial services, SaaS platforms, and high-throughput enterprise deployments, catch failures before they become customer-facing outages.
Why Kubernetes RabbitMQ Monitoring Is Different
A RabbitMQ cluster on bare metal or dedicated VMs has one primary adversary: its own workload. A RabbitMQ cluster on Kubernetes has several:
- Shared compute. Unless you've dedicated nodes, your RabbitMQ pods compete for CPU, memory, and disk I/O with whatever else Kubernetes schedules alongside them.
- Network abstraction layers. The Kubernetes networking stack adds pod-to-pod latency, CNI overhead, and a heartbeat layer that Raft consensus has to share bandwidth with.
- StatefulSet semantics. RabbitMQ on Kubernetes is deployed as a StatefulSet by the Cluster Operator, and StatefulSet pods do not automatically reschedule when a node fails — a fact that surprises operators at the worst possible moment.
- Persistent volume coupling. Local-path or node-local storage means a rescheduled pod may not be able to recover its previous state. Network-attached storage introduces its own I/O variability.
Every one of these adds a class of failure that would be impossible on dedicated infrastructure. Your monitoring has to account for all of them.
1. Raft Replication Health on Quorum Queues
If you're running quorum queues — and in 2026, if you're on RabbitMQ 4.x, you should be — Raft replication health is the single most important metric on your dashboard. If you haven't yet, see our guide on migrating to quorum queues. Quorum queues work by having a leader and followers participate in Raft consensus, writing every message to a majority of nodes before acknowledging the publisher. When Raft degrades, everything downstream of it degrades with it.
On Kubernetes specifically, Raft pressure compounds fast. The consensus protocol requires reliable low-latency communication between nodes, consistent disk write performance, and enough CPU headroom to process log entries. When any of those constraints tightens — and on Kubernetes, they tighten more often than on bare metal — Raft starts falling behind and queues start becoming unresponsive.
What to Actually Monitor
- rabbitmq_queue_messages broken down by leader vs follower state.
- rabbitmq_raft_log_commit_index and rabbitmq_raft_log_last_applied — the delta between these is your Raft lag.
- Leader election frequency. A healthy cluster has stable leaders; if leaders are flapping, something upstream (network, disk, CPU) is unhealthy.
- Per-queue Raft state via the management API: follower status, snapshot index, disk usage.
Alert Thresholds
Alert if a follower's Raft log trails the leader by more than a few thousand entries for more than 30 seconds. Alert if leader elections happen more than once per hour on a stable cluster. And alert immediately if any quorum queue has lost quorum — the broker will tell you through its metrics, and you want to know within seconds, not minutes.
Why This Matters More on Kubernetes
On Kubernetes, it is entirely possible for Raft to get into a state where the queue becomes unusable and the standard recovery procedures don't work cleanly. One production pattern we've seen: a quorum queue becomes stuck, rabbitmqctl delete_queue fails because the broker can't establish consensus to delete it, and restoring service requires killing pods in a specific order to force a new leader election. You do not want to discover this condition by customer complaint. You want to see the Raft lag climbing 10 minutes earlier.
2. Memory Watermark and Flow Control State
RabbitMQ has a built-in protection mechanism called the memory high-watermark. When a broker's memory usage crosses the watermark threshold, RabbitMQ blocks publishers to prevent the node from going out of memory entirely. This is by design — it's a failsafe — but on Kubernetes, the default watermark settings are often wrong.
Expert Insight: Kubernetes-Specific Watermark Tuning
"For Kubernetes, you should do the test with a 40% watermark and the net tick time of 60. We are letting more room to the Raft in order to do things, plus another room for the networking. The networking on RabbitMQ is not only between the nodes, it's also the networking of Kubernetes and the heartbeats of the pods. You are competing for resources."— Felipe Gutierrez, Senior Engineer, AceMQ
The default RabbitMQ memory watermark of 0.4 (40%) exists because on Kubernetes the broker is not the only thing using memory on the pod or the node. The Kubernetes kubelet, the CNI plugin, the container runtime, and any sidecar containers all consume memory that Raft and the broker need breathing room around. A watermark that makes sense on a dedicated VM often needs to be lowered on Kubernetes to preserve enough headroom for Raft operations and inter-pod networking.
What to Actually Monitor
- rabbitmq_resident_memory_limit_bytes and rabbitmq_process_resident_memory_bytes — the ratio between these is your current watermark utilization.
- rabbitmq_connections_blocked_total — how many publisher connections RabbitMQ is actively blocking.
- Flow control state per connection. If you see producers entering flow control, you have a pressure problem somewhere in the pipeline — check consumer prefetch tuning as a first step.
- Kubernetes pod memory usage vs pod memory limit — hitting the pod limit gets you OOMKilled, which is a much worse failure than hitting the RabbitMQ watermark.
Alert Thresholds
Alert at 80% of the configured watermark — at that point you're close to blocking publishers. Alert immediately on any connection being blocked. And alert at 90% of the Kubernetes pod memory limit — that's your last warning before the kubelet kills the pod.
3. Disk I/O Contention and Write Latency
This is the metric where Kubernetes bites the hardest. RabbitMQ quorum queues are disk-intensive by design — every write goes through the Raft log before being acknowledged. If the underlying storage is slow, variable, or contended, every producer in your system feels it.
On Kubernetes, the disk performance story is complicated by a specific failure mode we see repeatedly: RabbitMQ sharing a node with another I/O-heavy workload, typically PostgreSQL, Kafka, or Elasticsearch. When the neighbor workload spikes, RabbitMQ's Raft log writes slow down, follower lag grows, leaders get re-elected, and the cluster appears to mysteriously degrade for reasons the operator can't immediately trace.
Expert Insight: The I/O Contention Pattern
"If you're still competing with Postgres on the same node, Raft is not just memory and the quorum — it also requires disk. If you have heavy usage on Postgres at the same time, that will trigger the same discussion I'm having with you guys. You should separate those, or at least reduce the number of RabbitMQ nodes so you're not competing for resources."— Felipe Gutierrez, Senior Engineer, AceMQ
What to Actually Monitor
- rabbitmq_io_write_time_seconds_total and rabbitmq_io_write_bytes_total — write latency trends are your early warning.
- Disk write latency histograms at the node level (Kubernetes node_exporter provides these).
- IOPS consumed by RabbitMQ pods vs IOPS consumed by everything else on the same node.
- Disk queue depth — if this stays consistently above 1–2, you are waiting on disk.
- Persistent volume fill rate — quorum queue disk usage grows with message volume and log retention.
Alert Thresholds
Alert on p99 disk write latency exceeding 50ms on any RabbitMQ node. Alert on persistent volume utilization above 75%. And alert on disk queue depth persistently above 2 for more than a minute.
Architectural Recommendation
Wherever possible, dedicate Kubernetes nodes to RabbitMQ using node taints and tolerations. Run stateful I/O-heavy workloads on their own node pool. If that's not feasible, use separate persistent volumes on separate physical disks for RabbitMQ vs its neighbors — the cost of an additional disk is trivial compared to the cost of a production outage caused by I/O contention.
4. Pod Scheduling, Restart Count, and StatefulSet Behavior
Here is the Kubernetes-specific gotcha that catches operators off-guard: RabbitMQ is deployed as a StatefulSet by the official Cluster Operator, and StatefulSet pods do not automatically reschedule to a different node when their current node fails. This is by design — StatefulSet provides stable identity and storage — but the operational consequence is that a node failure can take a RabbitMQ pod offline for as long as the node is down.
This is not a theoretical concern. We have walked clients through incidents where a Kubernetes node went down, the RabbitMQ StatefulSet pod was stranded, and the application saw extended unavailability because the operator expected automatic rescheduling behavior that Kubernetes simply does not provide for StatefulSets.
What to Actually Monitor
- Pod restart count per RabbitMQ pod, tracked over time. A pod with 10+ restarts in a day is telling you something.
- Pod scheduling state — specifically, any RabbitMQ pod in Pending, Unschedulable, or ContainerCreating for more than 60 seconds.
- Node status for the nodes hosting RabbitMQ pods. A NotReady node is a ticking clock on a pod outage.
- StatefulSet readiness — how many of the desired replicas are currently Ready.
- Liveness and readiness probe failure rates. Probe failures under load usually indicate the broker is overloaded before it fully fails.
Alert Thresholds
Alert immediately on any RabbitMQ StatefulSet replica dropping below desired count. Alert on any pod restart. Alert on any node hosting a RabbitMQ pod going NotReady. These are all actionable conditions — you want a human looking at them, not an auto-healing mechanism that may or may not work on StatefulSets.
Architectural Note
If you need faster automatic recovery from node failure than the default StatefulSet behavior provides, you have a few options: use pod disruption budgets combined with node drain automation, deploy RabbitMQ across multiple Kubernetes availability zones so a zone failure doesn't take the cluster down, or — in some cases — run the broker on dedicated VMs outside Kubernetes entirely. The right choice depends on your recovery-time objective and your Kubernetes operational maturity.
5. Network Heartbeats and Inter-Node Latency
RabbitMQ nodes maintain distributed Erlang communication with each other, and the health of that communication determines whether the cluster remains a cluster. On Kubernetes, the networking layer introduces additional latency variability that can cause false-positive node failures — the broker thinks a peer is down because a heartbeat missed its window, when in reality the Kubernetes CNI was just momentarily slow.
The default Erlang net_ticktime of 60 seconds is often too aggressive for Kubernetes environments, particularly those with overlay networks or cross-zone traffic. Tuning this is one of the first things we do when hardening a Kubernetes RabbitMQ cluster for production.
What to Actually Monitor
- Network partition events — RabbitMQ logs these, and they should be alerted on immediately.
- Inter-pod latency between RabbitMQ pods (measurable via a simple ping sidecar or service mesh metrics).
- Erlang distribution buffer utilization — rabbitmq_erlang_distribution_buffer_busy tells you when inter-node communication is backing up.
- TCP connection state distribution at the node level. A growing count of TIME_WAIT or CLOSE_WAIT connections can indicate networking issues brewing.
- CNI-level metrics if available (Calico, Cilium, etc.) — packet drops between RabbitMQ pods are a canary.
Alert Thresholds
Alert on any network partition event. Alert on p99 inter-pod latency exceeding 10ms between RabbitMQ pods. Alert on Erlang distribution buffer utilization above 50%.
Tuning Recommendation
On Kubernetes, increase net_ticktime from the default 60 seconds to 120 seconds as a baseline. This gives the CNI layer more room to breathe and dramatically reduces false-positive cluster partition events under normal network jitter. Longer tick times mean slightly slower detection of truly failed nodes, but the trade-off overwhelmingly favors stability in Kubernetes environments.
Putting It All Together: Your Alerting Philosophy
Monitoring these five areas is not about creating more alerts — it's about creating the right alerts. In our experience, the operators who get paged the least are not the ones monitoring the least; they are the ones monitoring the right things with the right thresholds.
A sensible tiering for a Kubernetes RabbitMQ cluster:
- P1 (page immediately): Lost quorum on any queue. Node NotReady. StatefulSet replicas below desired. Any publisher connection blocked. Network partition detected.
- P2 (page during business hours): Raft follower lag growing persistently. Memory watermark above 80%. Disk latency p99 above 50ms. Pod restart. Leader election flap.
- P3 (dashboard only, review weekly): Capacity trends. Disk fill rate. Connection count growth. Queue depth growth.
Everything else is noise. If your on-call rotation is being paged for things that don't fit in P1 or P2, your alerts are misaligned with what actually matters.
Practical Takeaway
Kubernetes is a fine platform for RabbitMQ — when you monitor the things that Kubernetes makes different. Bare-metal monitoring mental models do not transfer cleanly. The five metric categories above are what we monitor on every Kubernetes RabbitMQ engagement we support, and they are what we recommend every team running RabbitMQ on Kubernetes instrument from day one.
If you inherit a Kubernetes RabbitMQ cluster and want to assess its monitoring posture quickly, check these five in order:
- Is Raft replication health visible per queue? If not, you're flying blind on quorum.
- Is the memory watermark set for Kubernetes (closer to 0.4 than 0.8)? If not, you're one traffic spike from blocked publishers.
- Is disk latency per RabbitMQ node monitored and alerted? If not, you can't diagnose I/O contention when it strikes.
- Is StatefulSet replica count and pod scheduling state alerted? If not, a node failure will blind-side you.
- Is net_ticktime tuned for your Kubernetes environment? If it's still the default, you're getting false-positive partitions.
Any cluster that answers "no" to two or more of those is one operational incident away from discovering exactly why the answer needed to be "yes."
The RabbitMQ monitoring checklist for SREs
If you only read one section of this post, read this one. Everything above explains why each signal matters; this is the list you put in the runbook. Enable the rabbitmq_prometheus plugin (built in since 3.8), scrape every node, and import the official Grafana dashboards. Then watch four groups of metrics.
Cluster. Every node reporting (up on each scrape target), no partitions (the management API health check or rabbitmqctl cluster_status, see the network partition section above), Raft leader stability on quorum queues (rabbitmq_raft_term_total should be flat; a rising term means elections are happening), and both resource alarms clear (rabbitmq_alarms_memory_used_watermark and rabbitmq_alarms_free_disk_space_watermark should be 0). Track raw headroom too: rabbitmq_disk_space_available_bytes and rabbitmq_process_resident_memory_bytes against rabbitmq_resident_memory_limit_bytes.
Queues. Depth (rabbitmq_queue_messages_ready), unacknowledged messages (rabbitmq_queue_messages_unacked), consumer count (rabbitmq_queue_consumers) and consumer utilisation (rabbitmq_queue_consumer_utilisation, where a low value means consumers are idle waiting on the broker and a value near 1 means the broker is waiting on consumers), redelivery rate (rabbitmq_queue_messages_redelivered_total), and dead-letter rate. Depth alone is not a problem; depth that keeps climbing while consumer count is steady is.
Throughput and latency. Publish and deliver rates (rabbitmq_queue_messages_published_total and rabbitmq_queue_messages_delivered_total, rated over a window) and publisher confirm latency. The broker does not export confirm latency directly, so measure it in the publishing application or with a canary publisher. A gap between publish rate and deliver rate that does not close is the earliest capacity signal you will get.
Platform. Pod restarts (kube_pod_container_status_restarts_total for the RabbitMQ StatefulSet), persistent volume latency from your storage class or node exporter, and CPU throttling (container_cpu_cfs_throttled_periods_total against container_cpu_cfs_periods_total). Throttling above a few percent on a broker pod shows up as confirm latency long before anything in RabbitMQ itself looks wrong. The pod scheduling and disk I/O sections above explain the mechanics.
The eight alerts we consider mandatory, with starting thresholds. Tune them to your workload; the point is to have a warning before the page.
| Metric | Warning | Page |
|---|---|---|
Node down (up == 0) | 1 node for 2 min | 2+ nodes, or 1 node for 10 min |
| Network partition detected | any, immediately | any lasting 5 min |
rabbitmq_raft_term_total rate | more than 1 election in 10 min | more than 3 in 10 min |
rabbitmq_alarms_memory_used_watermark | memory above 70% of limit | alarm set (publishers blocked) |
rabbitmq_disk_space_available_bytes | below 20% free | below 2 × the disk free limit |
rabbitmq_queue_messages_ready | growing for 15 min | above your agreed max depth for 5 min |
rabbitmq_queue_consumer_utilisation | below 0.6 for 10 min | below 0.3 with depth growing |
| Pod restarts | 1 in 1 hour | 3 in 1 hour, or CrashLoopBackOff |
Two things this table does not cover. First, unacked messages rising with a flat consumer count usually means a consumer is hung or prefetch is too high; alert on it per critical queue rather than globally. Second, dead-letter rate is workload specific, so alert on a change from baseline rather than a fixed number.
Wire the warning tier to a channel someone reads during the day and the page tier to on-call. If you are building the cluster from scratch, our Kubernetes deployment playbook covers the resource requests, storage classes, and anti-affinity rules that keep most of these alerts quiet. When one of them fires anyway, the troubleshooting guide maps each symptom to a fix.
Need Help Hardening Your Kubernetes RabbitMQ Cluster?
AceMQ runs production RabbitMQ on Kubernetes for enterprise clients across financial services, SaaS, healthcare, and Fortune 1000 operations. We build the monitoring stack, tune the broker for Kubernetes-specific failure modes, validate your failover procedures, and stay on retainer through production to handle the incidents you hope you never see.
If your team is struggling with RabbitMQ stability on Kubernetes — or planning a greenfield Kubernetes deployment and wants to get the monitoring right from the start — talk to our team about a RabbitMQ health assessment. We'll audit your current monitoring posture, identify the gaps, and build you a plan to close them. If you need RabbitMQ commercial support or RabbitMQ consulting, that's exactly what we do.
FAQ
How should SREs monitor RabbitMQ clusters?
Enable the rabbitmq_prometheus plugin on every node, scrape it alongside kube-state-metrics and cAdvisor, and import the official Grafana dashboards. Alert on four groups: cluster health (node up, partitions, Raft term changes, memory and disk alarms), queue health (ready, unacked, consumer count and utilisation, redeliveries), throughput (publish versus deliver rate, confirm latency measured at the publisher), and platform health (pod restarts, PV latency, CPU throttling). Use a warning tier that goes to a channel and a page tier that goes to on-call. The table above gives starting thresholds for the eight alerts we treat as mandatory.
What RabbitMQ metrics indicate capacity issues?
The earliest signal is publish rate consistently exceeding deliver rate. Next comes rabbitmq_queue_messages_ready growing while rabbitmq_queue_consumers stays flat, which means you need more consumers or faster ones. rabbitmq_queue_consumer_utilisation near 1 means consumers are the bottleneck; near 0 means the broker is. On the broker side, resident memory climbing toward rabbitmq_resident_memory_limit_bytes, rabbitmq_disk_space_available_bytes falling, and CPU throttling on the pod all point to the node being undersized. Memory or disk alarms firing means you are already out of capacity and publishers are being blocked.
What causes RabbitMQ cluster instability?
In our experience the causes are, in rough order: network partitions from node pressure or misconfigured pod disruption budgets; memory alarms triggered by deep queues or unbounded connection counts; disk contention on shared persistent volumes that makes Raft writes slow enough to trigger leader elections; StatefulSet pods rescheduled onto nodes where their volume cannot follow; and version drift between nodes during a rolling upgrade. Almost all of these show up first as a rising rabbitmq_raft_term_total or a pod restart, which is why those two alerts sit at the top of the table. Each cause has its own section above.
Do we need commercial monitoring for RabbitMQ?
No. The rabbitmq_prometheus plugin, Prometheus, and the official Grafana dashboards are free, well maintained, and enough for most estates, including large ones. What they do not give you is a person who reads them at 3 a.m., knows which of the eight alerts is the real one when three fire together, and has fixed it before on your version. That is what a support contract adds. We cover 3.8.x through 4.x with a 15-minute P1 response, so the value is in the response, not in the dashboards. Keep your own monitoring; bring us in for the interpretation and the fix.
Monitoring is step six of eight in the RabbitMQ on Kubernetes guide, which takes the Operator, storage, networking, requests and limits and rolling updates in order.
Go deeper on RabbitMQ
- GuideThe RabbitMQ Performance Tuning GuideRead the guide
- GuideThe RabbitMQ Streams GuideRead the guide
- GuideThe RabbitMQ Reliability Guide: Ten Failure Patterns and Their FixesRead the guide
- GuideThe RabbitMQ Disaster Recovery GuideRead the guide
- GuideThe RabbitMQ Clustering and Sizing GuideRead the guide
- GuideThe RabbitMQ on Kubernetes GuideRead the guide
- GuideThe RabbitMQ Migration GuideRead the guide
- GuideThe RabbitMQ Security and Hardening GuideRead the guide
- GuideThe RabbitMQ Monitoring and Alerting GuideRead the guide
- ComparisonManaged RabbitMQ Options ComparedSee the comparison
- ComparisonMessage Broker Support Options ComparedSee the comparison
- ResearchWhat Breaks in Production RabbitMQ: 145 Support Tickets, 2023 to 2026Read the research
- ResearchRabbitMQ in Production 2026: What 22 Assessed Estates Actually RunRead the research
- ResearchThe RabbitMQ CVE Register, 2026 EditionRead the research
