Kafka on Kubernetes: EKS vs AKS vs GKE

Kafka on Kubernetes works, but it is a stateful workload wearing a stateless platform's clothes. Brokers carry identity, data and partition leadership, so pod churn is never free. EKS, AKS and GKE differ mainly in storage class semantics, zone-redundant disk options and load balancer behaviour — the Kafka-side design is identical on all three.

The provider choice is the smaller half of the decision. Most of what goes wrong with Kafka on Kubernetes goes wrong the same way on every cloud: a broker comes back with a different identity, a replica set collapses into one zone, or a client bootstraps successfully and then cannot reach a single broker.

If you are weighing the same question for RabbitMQ, we covered running RabbitMQ on EKS, AKS and GKE separately — the storage reasoning rhymes, the failure modes do not.

Three properties separate Kafka on Kubernetes from the deployments Kubernetes was designed around.

Brokers have identity. A broker is not interchangeable with its replacement. It owns a node ID, a set of partition replicas, and leadership for some of them. Kubernetes wants to treat a pod as fungible; Kafka does not. That is why brokers run as StatefulSets with stable network identities and per-pod volumes rather than as a Deployment.

Brokers own data. Every broker's log directory holds partition segments that are expensive to rebuild. A pod that returns without its old volume is not a restart, it is a full replica rebuild — pulled from the leader across the network, competing with live traffic for the same bandwidth. On a large broker that is hours, not seconds.

Pod churn triggers Kafka work. Node autoscaling, spot reclamation, a rolling upgrade, an OOM kill: each takes a broker out, moves leadership, and shrinks the in-sync replica set while it is gone. Consumer groups feel it too — what triggers a Kafka rebalance covers the consumer side of the same event.

So a Kubernetes cluster tuned for fast, aggressive pod movement is actively hostile to Kafka. You want the opposite: PodDisruptionBudgets that refuse to take a second broker down while the first is still catching up, and rolling updates that wait for ISR to recover rather than for a readiness probe to go green.

One thing has got easier. Apache Kafka 4.0 removed ZooKeeper mode entirely and runs KRaft only, so there is no separate ensemble alongside the brokers. That is one less StatefulSet — though controllers are still stateful processes needing their own volumes and disruption budget.

Strimzi is the main open-source Kafka Kubernetes operator and the one most estates end up on. Check the current feature set against the docs for the version you are installing — custom resource shapes have changed across API versions — but as of writing it provides:

• Cluster Operator for brokers and controllers, plus Kafka Connect, MirrorMaker 2 and the HTTP Bridge.
• Topic Operator and User Operator, so topics and ACLs are declared as KafkaTopic and KafkaUser resources instead of created by a script nobody can find later.
• Node pools (KafkaNodePool), giving controllers and brokers different roles, storage and instance shapes inside one logical cluster.
• Drain Cleaner, which intercepts pod evictions so the operator moves brokers deliberately rather than letting the node drain do it.
• Cruise Control integration for partition rebalancing, and a tieredStorage property for offloading older segments to object storage.

The commercial distributions — Confluent Platform's operator, Red Hat's build of Strimzi — wrap the same ideas with a support contract. Choosing between them is a procurement question.

What an operator buys you is the boring, error-prone parts: ordered rolling restarts that respect ISR, certificate rotation, and generating advertised.listeners correctly for each listener type. Running Kafka on Kubernetes with raw StatefulSets is possible and we have seen it done. It is also where most of the incidents come from.

This is where the providers actually diverge. The Kafka configuration does not change; the volume underneath it does.

Kafka has had a rack concept since long before anyone ran it on Kubernetes, and it maps onto cloud availability zones almost perfectly.

Set broker.rack on each broker to the zone it is running in. Kafka's replica assignment then spreads replicas so that, in the Apache documentation's words, a partition will span min(#racks, replication-factor) different racks. With three zones and replication factor 3, every partition has a replica in each zone, and losing a zone costs you one replica per partition rather than all of them.

For Kafka on Kubernetes you do not set this by hand. Strimzi reads a node label — conventionally topology.kubernetes.io/zone — and injects the value as broker.rack for each broker pod. That requires the operator to read cluster-scoped Node objects, so RBAC has to allow it; a rack configuration that silently does nothing is usually a missing ClusterRoleBinding.

Two traps worth naming:

Uneven brokers per zone skews replica placement. Apache's own guidance is blunt: racks with fewer brokers get more replicas, so they use more storage and spend more on replication. Six brokers as 3/2/1 across zones is worse than 2/2/2, even though the totals match.

Rack awareness is not zone anti-affinity. broker.rack tells Kafka how to place replicas. It does not tell Kubernetes how to place pods. You need both: topologySpreadConstraints (or pod anti-affinity) so the scheduler distributes brokers across zones, and broker.rack so Kafka knows what that distribution means.

Once racks are set, the same labels let consumers fetch from the closest replica. Set client.rack on consumers and replica.selector.class to org.apache.kafka.common.replica.RackAwareReplicaSelector on the brokers, and a consumer reads from an in-zone follower rather than crossing a zone to reach the leader. On a chatty cluster that is a real cross-zone transfer line item, removed by one configuration change.

This is the classic failure. It looks like a network problem, it is a metadata problem, and it behaves identically on all three clouds.

Kafka clients do not keep talking to whatever address you gave them. They connect to a bootstrap server, ask for cluster metadata, and receive a list of broker addresses — the values in each broker's advertised.listeners. From that point on they connect directly to those addresses. The bootstrap connection succeeded; the ones that matter never do.

Inside Kubernetes the natural addresses are pod IPs and headless-service DNS names. Both are meaningless outside the cluster. So the symptom is a client that authenticates, fetches metadata, then times out against hosts it cannot resolve — while kubectl exec into any broker shows a perfectly healthy cluster.

The fix is to give each listener an advertised address that is correct for the clients using that listener, which means separate listeners:

• Internal listener — headless service DNS, for in-cluster producers, consumers and inter-broker traffic.
• External listener — per-broker addressable endpoints. Not one load balancer in front of all brokers: clients must reach a specific broker, because partition leadership is per-broker. Operators implement this as a load balancer or node port per broker, plus a bootstrap endpoint.

advertised.listeners defaults to null and listeners defaults to PLAINTEXT://:9092, so anything here is a deliberate override. Let the operator generate it. Hand-rolling advertised addresses is how a Kafka on Kubernetes cluster works right up until the day it scales.

Two related points: put the external listener behind TLS and real authentication before it exists at all, and remember a per-broker load balancer set is a per-broker bill on every cloud. That cost is a common reason teams end up back at the managed-service conversation.

Honest answer: more often than engineers like to admit, and less often than vendors claim.

Managed wins when Kafka would be the first stateful workload on your cluster, when nobody on the team has taken a broker through a disk-full incident, or when the engineering time you would spend on upgrades and rebalances is worth more elsewhere. The first-party options differ by cloud: Amazon MSK offers Standard and Express brokers (Express replicates across three AZs and is only available in a three-AZ configuration) plus MSK Serverless; Google Cloud Managed Service for Apache Kafka is generally available; Azure has no first-party Apache Kafka service, so the comparison there is Event Hubs' Kafka protocol endpoint — which speaks Kafka 1.0 and later but is not a Kafka cluster and does not behave like one in every corner — or a third party such as Confluent Cloud.

Self-managed wins when you already run stateful workloads on Kubernetes competently, when you need broker-level configuration a managed tier will not expose, when data residency or network topology rules it out, or when your volume makes the managed premium genuinely large.

The decision rarely turns on the cloud provider. It turns on whether your team's Kubernetes maturity is real. On a recent payments engagement we ran a RabbitMQ upgrade using Kubernetes sidecar deployments and a staged migration to reach zero downtime — and that customer was evaluating managed options in parallel, which is the right instinct. Both questions deserve an answer before you commit.

Four signals separate a healthy Kafka on Kubernetes cluster from one quietly degrading:

• Under-replicated partitions. The single most useful number. A value that does not return to zero after a rolling restart means a broker is not keeping up.
• Volume utilisation per broker, not per cluster. Averages hide the one broker about to fill, and a full log directory takes a broker offline.
• Pod restart counts and eviction events. Correlate against under-replication. A node pool recycling nodes on a schedule you did not choose shows up here first.
• Consumer lag. The user-visible symptom of all of the above — diagnosing Kafka consumer lag covers reading it properly.

Tiered storage (KIP-405, production-ready as of Kafka 3.9) changes disk sizing considerably if retention is long: older segments move to object storage, brokers keep less locally, and replica rebuilds get shorter. Worth evaluating before buying another terabyte per broker.

Planning a Kafka deployment on EKS, AKS or GKE — or trying to work out whether you should be self-managing at all? AceMQ runs Kafka assessments that cover exactly this decision, including the storage, rack and listener design that most tutorials skip. See Apache Kafka support and consulting and our container and Kubernetes messaging services, or talk to an AceMQ engineer.

• Apache Kafka — Broker configuration reference
• Apache Kafka — Basic operations, balancing replicas across racks
• Apache Kafka — 4.0.0 release announcement (KRaft-only, ZooKeeper removed)
• Strimzi — Configuring Strimzi (storage, rack, node pools, tiered storage)
• AWS — Amazon MSK broker types
• Microsoft — Storage concepts in Azure Kubernetes Service
• Google Cloud — GKE persistent volumes and StorageClasses