Messaging on Kubernetes: Why RabbitMQ and Kafka Are Harder Than They Look

Messaging brokers like RabbitMQ and Kafka are stateful workloads running on a platform built for stateless ones — Kubernetes was designed around pods being disposable "cattle," while a broker's identity, replicated data, and quorum state make every pod a "pet." That mismatch is where most container messaging incidents start.

Kubernetes handles stateless web tiers extremely well: kill a pod, a new one starts, nothing is lost. Message brokers don't get that luxury. A RabbitMQ node or Kafka broker carries data, cluster membership, and — in RabbitMQ's case — quorum votes that the rest of the cluster is actively counting on. Treat it like a stateless pod and you get stuck terminations, split clusters, and messages that quietly stop flowing.

This is the general pattern behind the product-specific problems we've covered for RabbitMQ on Kubernetes and OpenShift and Kafka on EKS, AKS, and GKE. Choosing a cloud provider for either broker is its own question, covered in RabbitMQ on EKS vs AKS vs GKE. This piece is about the failure modes both brokers share simply by being stateful workloads on Kubernetes at all.

Kubernetes' entire scheduling model assumes pods are interchangeable. The scheduler can kill, reschedule, and recreate a pod on any node, and a Deployment's ReplicaSet doesn't care which physical node runs replica 3 — it just needs three healthy replicas somewhere. That's the "cattle, not pets" philosophy the whole container ecosystem is built on, and it's the right model for stateless services.

A message broker breaks that assumption in three ways. It has a durable identity — a Kafka broker's node ID or a RabbitMQ node's name is baked into cluster metadata, not interchangeable with a same-named replacement. It owns data on disk the cluster depends on, not a cache that can be dropped and rebuilt for free. And it participates in a quorum or replication protocol, so removing one node changes what the surviving nodes are allowed to do. A vanilla Kubernetes Deployment or bare pod doesn't know any of that — it just sees a container that stopped and starts another one.

That's why both brokers run as StatefulSets rather than Deployments. A StatefulSet gives each pod a stable, ordinal hostname (rabbitmq-0, kafka-broker-2) and a stable DNS name through a headless Kubernetes Service, and — critically — it creates and terminates pods in strict order, one at a time, waiting for each predecessor to be Ready before starting the next. That ordering is a deliberate control, not a default Kubernetes behavior, and it exists precisely because a messaging cluster can't tolerate the simultaneous node churn a stateless Deployment shrugs off. It's also why a message queue or message broker running as a bare Docker container on a single host — no k8s, no orchestration — is sometimes the right call for a small deployment: it sidesteps this whole category of problem by not having a scheduler in the loop at all.

Every stateful broker on Kubernetes is solving the same underlying problem: how do you keep a pod's identity and data attached to it across restarts, reschedules, and node failures, when the platform's default behavior is to treat all of that as disposable?

The StatefulSet's PersistentVolumeClaim, one per pod via volumeClaimTemplates, is the mechanism. Deleting a pod, or even scaling the StatefulSet down, does not delete its PVC by default — that's deliberate, so a rescheduled or restored replica reattaches to its old data instead of starting empty. Kubernetes 1.27 made that behavior configurable through persistentVolumeClaimRetentionPolicy, letting you set separate Retain/Delete rules for whenScaled versus whenDeleted — but retention is still the default, and for a broker it's almost always the setting you actually want.

None of this is automatic protection against every failure mode, though. It just gives Kubernetes the primitives — stable identity, stable storage, ordered lifecycle — that a stateful workload needs. Whether those primitives are configured correctly is a separate question, and it's the one that actually causes incidents.

Eviction timing is where things go wrong in practice. Kubernetes evicts pods for reasons that have nothing to do with the messaging cluster's internal state — node pressure, a cluster autoscaler reclaiming a node, a spot instance getting reclaimed, or a manual drain during a cluster upgrade. None of those triggers check whether a Kafka partition reassignment or a RabbitMQ quorum election happens to be mid-flight when they fire.

For Kafka, evicting a broker while it's the partition leader for in-flight writes forces a new leader election and shrinks the in-sync replica set until the replacement catches up. If the evicted broker also held a consumer group's coordinator role, consumers stall until a new coordinator is elected. Chain a second eviction before the first broker fully rejoins and rebuilds its replicas, and you can trigger a rebalance storm — consumer groups repeatedly reshuffling partition assignments instead of processing messages.

For RabbitMQ, quorum queues use Raft-based replication, so losing one node mid-election is survivable as long as a majority remains — but evict two nodes out of three at once and the survivor can't form quorum on its own. Queues on that cluster stop accepting new writes until a majority of nodes is back, even though the pod count in the namespace still shows activity.

Fixing this isn't a single Kubernetes setting — it's making sure Kubernetes never voluntarily evicts more nodes at once than the messaging protocol can tolerate, which is exactly what a PodDisruptionBudget is supposed to enforce (more on where that falls short below).

When a stateful pod is rescheduled onto a new node, its PersistentVolumeClaim has to detach from the old node before it can attach to the new one. For network-attached block storage — EBS, Azure Disk, Persistent Disk — that's not instantaneous. Kubernetes has to confirm the old node released the volume, the cloud provider's storage API has to process the detach request, and only then does the new node's attach succeed.

The failure teams hit most often is a Multi-Attach error: the new pod sits stuck in ContainerCreating because the volume is still marked attached to the old node — usually because the old pod wasn't cleanly terminated. A hard node failure, a force-deleted pod, or a kubelet that never got to report the volume as released before the node disappeared will all cause this. Because most cloud block storage only supports ReadWriteOnce access, Kubernetes can't just attach the volume to both nodes and sort it out later; it has to wait.

That wait can run minutes for a single volume, and minutes matter far more for a broker than for a stateless pod — its peers spend that entire window treating it as gone, which is exactly the eviction scenario in the section above. Sizing PVCs correctly, using a storage class with fast provisioning, and — as covered above — not force-deleting broker pods without understanding what state they're holding are the practical mitigations, not a Kubernetes setting that makes reattachment instant.

Not a network partition in the literal, severed-cable sense — but the practical effect can be close enough that it's worth planning for. A rolling update on a StatefulSet takes pods down one at a time, in reverse ordinal order, and waits for each replacement to report Ready before moving to the next. If the readiness probe only checks "is the process up" rather than "has this node rejoined the cluster and caught up on replication," Kubernetes will happily proceed to the next pod while the previous one is still resyncing — and now two nodes out of three are degraded at once, which for a quorum-based broker is functionally the same as losing a network link.

The other common trigger is a configuration change rolled out alongside the broker upgrade — a network policy or service mesh rule that the new pod doesn't inherit the same way its predecessors did, or a scheduling change that lands it in a different node pool with different egress rules. A pod that comes up clean but can't reach its peers on the inter-broker port looks identical, from the cluster's perspective, to an actual partition — even though nothing about the underlying network link failed.

PodDisruptionBudgets are the right tool, but they only cover part of the problem. A PDB limits voluntary disruptions — node drains, cluster-autoscaler scale-downs, and rolling updates initiated through the Kubernetes API. It does nothing for involuntary disruptions: a node that crashes, an out-of-memory kill, or a spot instance reclaimed without warning. Kubernetes' own documentation is explicit that a PDB "does not truly guarantee" the specified number of pods stay available — it can only stop Kubernetes itself from voluntarily taking more pods down than the budget allows.

Set a PDB too conservatively — maxUnavailable: 0, say, on a three-node quorum cluster — and you create a different problem: a node drain that can never complete, because Kubernetes refuses to evict the one pod standing between the budget and a violation. That's not a bug, it's the PDB doing exactly what it was configured to do. It just means the drain, and the cluster upgrade behind it, hangs until someone intervenes — a common enough incident pattern in its own right.

For a three-node quorum-based broker, the setting that actually works in practice is usually maxUnavailable: 1: it lets Kubernetes take one node down for maintenance while guaranteeing the remaining two can still form quorum, without blocking every drain indefinitely.

None of the failure modes above rule out running messaging on Kubernetes — they mean the platform's defaults aren't enough on their own. What consistently works in production:

• Use the official operator, not raw manifests. The RabbitMQ Cluster Operator and Strimzi for Kafka encode the ordering, pre-stop, and reconciliation logic that hand-rolled StatefulSet YAML leaves you to reinvent — and usually get wrong the first time.
• Set a PodDisruptionBudget that matches your quorum math, not a default copied from a stateless service — maxUnavailable: 1 on a three-node quorum cluster, not 2.
• Use pod anti-affinity across nodes and availability zones so a single node or zone failure can't take out a quorum majority in one event.
• Use a storage class with volumeBindingMode: WaitForFirstConsumer so volumes provision in the zone the pod actually lands in, and size them for a full replica rebuild, not just steady-state usage.
• Build readiness probes that check cluster health, not process health. A broker that's running but hasn't rejoined the cluster or caught up on replication should never report Ready.
• Slow rolling updates down deliberately during upgrade windows, and confirm quorum or in-sync replica counts between each pod restart instead of trusting the default rollout pace.

This is the same operational discipline regardless of which cloud runs the cluster — the Kubernetes primitives are identical on EKS, AKS, GKE, and OpenShift. What differs is whether someone configures them for a stateful broker instead of leaving the stateless defaults in place.

Sometimes the honest answer is no. Running RabbitMQ or Kafka on Kubernetes makes sense when your team already operates production Kubernetes clusters at a mature level — someone owns storage classes, PodDisruptionBudgets, and anti-affinity as a matter of course — and you want the broker to fit the same deployment and observability model as the rest of your cloud native stack.

It makes less sense when Kubernetes-hosted messaging would be the first genuinely stateful workload on the cluster, or when nobody on the team has taken a broker through a real quorum-loss or disk-full incident. In that case, a managed messaging service, or a broker deployed outside Kubernetes on dedicated VMs, trades some deployment convenience for a lot fewer 3 a.m. pages. The decision has less to do with which cloud provider you're on and more to do with whether your Kubernetes operational maturity is actually there yet.

Planning a RabbitMQ or Kafka deployment on Kubernetes — or debugging a cluster that's stuck mid-eviction right now? AceMQ's engineers handle stateful messaging operators in production, not just the stateless workloads Kubernetes was built for. See our Kubernetes and container messaging services, go deeper on RabbitMQ or Kafka specifically, or talk to an AceMQ engineer.

• Kubernetes — StatefulSets (identity, ordering, PVC retention)
• Kubernetes — Configuring PodDisruptionBudgets
• Kubernetes — Persistent Volumes and access modes
• RabbitMQ — Using the Cluster Operator on Kubernetes
• Strimzi — Configuring Kafka on Kubernetes