Overview
Kubernetes supports each minor release for about a year, so a cluster that skips upgrades falls out of community support quickly. The blockers are rarely Kubernetes itself. They are removed APIs in old manifests, operators that do not support the target release, and storage drivers tied to the cluster.
Challenge
Production clusters ran a minor release the project had stopped patching, and two earlier upgrade attempts were rolled back after Helm charts failed on removed APIs. A Strimzi-managed Kafka cluster and several other stateful workloads made the platform team reluctant to try again without a written plan.
Environment
Two kubeadm clusters on VMware vSphere and one AKS cluster, about 45 Helm releases, a Strimzi-managed Kafka cluster, CSI-backed persistent volumes, and Argo CD for delivery.
Approach
AceMQ inventoried every manifest and live object for deprecated and removed APIs, compared operator and driver versions with what each target release supports, and ordered the upgrade so stateful workloads moved last, after a rehearsal on a cloned cluster.
Solution
- Scanned manifests, Helm charts and live objects for APIs removed between the current and target releases
- Listed every operator, CSI driver and CNI plugin with the first version that supports the target release
- Upgraded Strimzi and the Kafka brokers first, in the order its compatibility matrix requires, before the control plane moved
- Rehearsed the control plane and node upgrades on a cloned staging cluster and recorded the timings
- Wrote a minor-by-minor upgrade plan with a rollback point and a named owner for each step
- Added a deprecated-API check to CI so a removed API fails a pull request instead of an upgrade
Outcome
The clusters moved to a supported release in two maintenance windows, and the platform team now plans one minor upgrade per quarter instead of avoiding them.