Overview
This is a typical engagement for a platform team that runs production on a managed Kubernetes service. The cluster sits on a minor version that upstream no longer patches, the provider's standard support window is closing, and the team would rather build a new cluster than upgrade in place across several minors. Most services are stateless. A RabbitMQ cluster, a PostgreSQL primary with replicas and a Redis cache are not.
Challenge
Persistent volumes belong to one provider's storage and usually to one zone, so state cannot be moved by copying a volume. Storage classes, ingress controllers, certificate issuers and secrets differ between the two clusters, and some of what the old cluster runs was installed by hand and appears in no manifest. A plan that says "redeploy and repoint DNS" has no answer for the data.
Environment
A managed Kubernetes cluster several minors behind, a larger set of stateless services, a RabbitMQ cluster, a PostgreSQL primary with streaming replicas, a Redis cache, an ingress controller that is being retired, and a team that deploys through Helm and a GitOps tool.
Approach
AceMQ and a partner Kubernetes specialist inventory what each namespace needs from its cluster and mark every item as moved, rebuilt or retired. The target cluster is designed first. One stateless service and one stateful service are then rehearsed end to end against a copy of production data, and the rollback for each is run during the rehearsal. The wave plan is written from what the rehearsal showed.
Solution
- Inventory of workloads, storage classes, persistent volumes, ingress, certificates, secrets, CRDs and anything installed by hand
- Target cluster design: version, node pools, storage classes, ingress or Gateway API, network policy, identity and observability
- Data-movement method per workload: a second RabbitMQ cluster with Shovel or Federation, a PostgreSQL replica promoted at cut-over, a Redis replica or snapshot restore
- Rehearsal against a copy of production data, with replication lag, queue depth and client reconnect behavior recorded
- Wave plan with go and no-go checks, traffic shifted in steps and a rollback point for every wave
- Decommission checklist for the old cluster and an upgrade calendar for the new one
Outcome
The engagement delivers the written inventory, a target design, a rehearsed method for each stateful workload with its rollback, a wave plan with the checks for each step, and a decommission checklist. The customer's engineers can run the waves themselves, or AceMQ can run them alongside them.
Technologies
Kubernetes
RabbitMQ
PostgreSQL
Redis
Amazon EKS
Prometheus
OpenShift
