Overview
This is a typical engagement for a platform team that runs several GKE clusters across development, staging and production. Clusters upgrade automatically, a node pool upgrade has stalled before, and nobody has written down which release channel each environment should use or when upgrades may run. AceMQ engineers, with a partner Kubernetes specialist where needed, review the settings and write the policy.
Challenge
The release channel decides which version a cluster is offered, and the maintenance window and exclusions decide when it arrives. Google documents that deprecation insights pause auto-upgrade to the next minor version only when use of a removed API is detected, and that detection is not guaranteed. Node pool surge upgrades respect PodDisruptionBudgets and graceful termination for up to one hour, after which Pods are forcefully evicted, so a tight budget turns an upgrade into a forced eviction.
Environment
Several GKE Standard clusters on different release channels, one cluster on the deprecated no-channel option, a default maintenance policy, Helm releases deployed from CI, and workloads with PodDisruptionBudgets that were copied between services.
Approach
AceMQ engineers read each cluster's channel, maintenance window, exclusions, node pool upgrade settings, PodDisruptionBudgets and deprecation insights, and compare them with Google's documented behavior. They propose a channel per environment, a window and exclusion rules within Google's limits, and a node pool strategy, then rehearse the next minor version on a non-production cluster.
Solution
- Channel per environment, with the reason, and a plan to enroll any cluster that is on the no-channel option before Google removes it
- One maintenance window per cluster and exclusion rules that stay inside Google's limits on count and length
- Node pool upgrade settings per pool: surge values, or blue-green where the rollback is worth the extra capacity
- PodDisruptionBudget and replica review so drains finish well inside the one-hour limit
- Deprecated API findings from deprecation insights and a scan of manifests, with the fix for each caller
- Rehearsal of the next minor upgrade on a non-production cluster, with a rollback point and an owner for every step
Outcome
The engagement delivers a written upgrade policy with a channel for every cluster, the maintenance windows and exclusions, node pool settings, the PodDisruptionBudgets to change, the deprecated API findings and a rehearsal plan for the next minor version. The customer's team runs the upgrades from the policy, or AceMQ can run them alongside it.