Running RabbitMQ on Kubernetes & OpenShift: A Deployment Playbook

RabbitMQ on Kubernetes or virtual machines: how to decide

RabbitMQ runs well on both. The deciding factors are who operates the platform and what the storage underneath looks like.

Who runs it. If a platform team already runs stateful workloads on Kubernetes, with an operator, PodDisruptionBudgets and StatefulSet upgrades as daily routine, RabbitMQ is one more StatefulSet. If your infrastructure team runs VMs well and Kubernetes is only where the stateless apps live, putting the broker there means the messaging team learns Kubernetes failure modes on a production system.

Storage. Quorum queues and streams fsync on every batch of writes. On a VM with local NVMe that is sub-millisecond. On Kubernetes the default is a network-attached PersistentVolume, and every fsync crosses the storage network. We typically see 2 to 5 ms per fsync on cloud block storage, which shows up directly as publisher confirm latency on quorum queues. Local PV provisioners exist but pin the pod to a node. Classic vs quorum queues covers why quorum queues care about this more than classic queues do.

Upgrades. Kubernetes does a rolling StatefulSet restart, one pod at a time, with the operator waiting for quorum before each step. On VMs you upgrade in place, node by node, on your own schedule. The Kubernetes version is more automated and less forgiving of a broken assumption.

Failure domains. On Kubernetes, nodes get drained, pods get evicted and the autoscaler reclaims capacity without asking you. A PDB and pod anti-affinity are the only things between a routine drain and a lost quorum. VMs fail less often and on schedules you control, but they fail harder, with no scheduler to restart the broker somewhere else.

Networking. Pods get hostnames from the headless service and IPs that change on every restart. Clients connect through a Service or ingress and must reconnect cleanly, because they will be reconnected. Anything that pins a broker IP breaks. On VMs, hostnames stay stable for years and client reconnect bugs stay hidden.

Licensing. Tanzu RabbitMQ, the commercial distribution, is packaged first for Kubernetes, with a VM image alongside. Open-source RabbitMQ runs anywhere.

Our recommendation: VMs when the team is small and latency-sensitive quorum queue workloads dominate. Kubernetes when there is a platform team, an operator in use, and the rest of the estate already lives there. Across our engagements the split is roughly even, and the Kubernetes deployments that go badly are almost always the ones adopted because Kubernetes was available, not because anyone was ready to operate stateful workloads on it.

Operator, Helm chart, or hand-rolled StatefulSet

The RabbitMQ Cluster Kubernetes Operator is the upstream-maintained option and the one we default to. You declare a RabbitmqCluster resource and the operator creates the StatefulSet, the headless service for peer discovery, the client Service, the Erlang cookie and default user Secrets, and the server configuration. It handles rolling restarts with a preStop hook that waits for quorum-plus-one before taking a node down, and it wires TLS from a Secret you point it at. It does not create a PodDisruptionBudget on its own; add one with maxUnavailable: 1. Alongside it, the Messaging Topology Operator turns users, vhosts, permissions, queues, exchanges, bindings, policies, shovels and federation upstreams into CRDs, so topology lives in Git next to the cluster definition.

A minimal production-shaped cluster:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: orders
spec:
  replicas: 3
  resources:
    requests:
      cpu: "2"
      memory: 8Gi
    limits:
      cpu: "2"
      memory: 8Gi
  persistence:
    storageClassName: fast-ssd
    storage: 100Gi
  rabbitmq:
    additionalConfig: |
      cluster_partition_handling = pause_minority
      vm_memory_high_watermark.relative = 0.6
      disk_free_limit.absolute = 5GB

Keep requests equal to limits. The operator derives RabbitMQ's memory override from the pod's memory setting, so the high watermark and the cgroup limit agree. Add spec.affinity with pod anti-affinity across nodes, or zones if you have them.

Community Helm charts template a StatefulSet plus the services, config and secrets in one helm install. They are fine for getting a cluster up. They do nothing after install: no ordered rolling upgrade that waits for quorum, no topology management, and every helm upgrade rewrites the StatefulSet and hopes. If you standardise on Helm for everything, run the operator and manage the RabbitmqCluster resource with a chart instead.

A hand-rolled StatefulSet is defensible in two cases: a cluster where you are not allowed to install CRDs or cluster-scoped operators, or a team that already has a hardened StatefulSet pattern for other stateful systems and wants RabbitMQ to match it. What you take on is everything above: peer discovery configuration, the preStop drain hook, probe tuning, secret rotation and the upgrade choreography. The recurring bug we inherit is an upgrade that restarted two pods at once.

The Kubernetes failure modes we get called about

A drain took the quorum leader, and the PDB let it. Symptom: publishers block on confirms for 10 to 30 seconds during a routine node upgrade, or a queue reports no leader because two of three replicas are down. Cause: no PDB, or one that allows more than one pod unavailable, combined with a cluster upgrade that drains nodes faster than quorum queues re-elect and catch up. Fix: a PDB with maxUnavailable: 1, pod anti-affinity so replicas never share a node, and the operator's preStop hook (or your own rabbitmq-upgrade await_online_quorum_plus_one) so the drain waits.

PV reattach after a node failure. Symptom: after a node dies, its RabbitMQ pod sits in ContainerCreating with a Multi-Attach error or a volume node affinity conflict for six minutes or more. Cause: zone-bound block storage can only attach in its own zone, and the dead node's attachment is not released until the node controller gives up on it. Fix: the cluster stays up on two of three replicas if anti-affinity spread them, so do not fight it. Use a WaitForFirstConsumer storage class so the volume lands in the pod's zone, and treat the reattach window as a known cost of network storage.

OOM-killed before flow control. Symptom: a pod restarts with exit code 137 and no RabbitMQ error in the log; the memory alarm never fired. Cause: the container memory limit sits below the point where the high watermark would engage, either because RabbitMQ read host memory instead of the cgroup limit, or because the watermark leaves no headroom for the Erlang VM's allocators and binary heap. Fix: requests equal to limits, a watermark that leaves 20 to 30 percent below the limit, and rabbitmq-diagnostics memory_breakdown to confirm the total RabbitMQ sees matches the pod.

Liveness probe restarts during recovery. Symptom: after an unclean restart a node with large queues goes into CrashLoopBackOff, each attempt getting further through the index rebuild before the kubelet kills it. Cause: a liveness probe running rabbitmq-diagnostics check_running or similar fails while the node replays Mnesia or Khepri and rebuilds queue indexes, which on a big node takes longer than the probe's failure threshold. Fix: a startup probe with a generous failureThreshold, a readiness probe on port 5672 only, and no aggressive liveness probe at all. The operator ships without a liveness probe for exactly this reason.

A DNS or headless service change broke peer discovery. Symptom: pods start but each forms a cluster of one, or refuse to boot because the data directory belongs to a node with a different name. Cause: node names derive from the pod hostname and the headless service, so renaming the service, the namespace or the cluster DNS domain changes every node name and orphans the on-disk data. Fix: do not rename. If a rename is unavoidable, treat it as a migration to a new cluster: export definitions, shovel or drain the queues, and cut clients over.

For the wider list of what breaks when messaging lands on Kubernetes, see what breaks with messaging on Kubernetes. For what to watch so these arrive as alerts rather than tickets, see RabbitMQ Kubernetes monitoring. If you would rather have someone who has fixed these before, see our Kubernetes and containers consulting and RabbitMQ support.