Overview
A shared Kubernetes platform built for stateless services sets defaults that hurt databases and brokers: shared node pools, general-purpose storage classes and eviction policy that treats every pod alike. Agreeing the platform contract for stateful workloads first is cheaper than finding the gaps in an incident.
Challenge
The customer planned to move a Kafka cluster and several PostgreSQL databases from virtual machines onto its existing Kubernetes platform. The platform had no storage class with a guaranteed IOPS floor, no taint scheme for dedicated node pools, and no policy on what happens to a volume when a StatefulSet is deleted.
Environment
Azure AKS across three zones, Strimzi for Kafka, a PostgreSQL operator for the databases, Premium SSD v2 volumes under evaluation, and Prometheus with Grafana.
Approach
AceMQ ran design sessions with the platform and database teams, benchmarked candidate storage classes with fio against the fsync patterns of Kafka and PostgreSQL, and wrote the platform contract as policy the cluster enforces rather than a wiki page.
Solution
- Benchmarked three storage classes with fio and chose volumes sized for the measured fsync latency and IOPS
- Created dedicated tainted node pools for brokers and databases, with zone-aware topology spread
- Set PodDisruptionBudgets and anti-affinity per workload, and tested a node drain against each
- Set PVC retention and the storage class reclaim policy so that deleting a StatefulSet cannot silently delete data
- Added Kyverno policies that reject stateful workloads without resource requests, limits and a disruption budget
- Wrote upgrade and failure runbooks for the operators and agreed who owns each one
Outcome
Kafka and the first three databases moved onto the platform on schedule, and each passed a scripted node-drain test before taking production traffic.
Technologies
Kubernetes
KafkaStrimzi
PostgreSQL
dbt
Docker
