Overview
This is a typical engagement for a platform team whose mesh runs an Istio minor that the project is about to stop supporting. Istio supports a minor release until 6 weeks after the minor two releases newer ships, so the team needs an upgrade it can schedule. AceMQ engineers and a partner service mesh specialist plan a revision-based canary upgrade, which Istio documents as safer than an in-place upgrade and recommends.
Challenge
Installing a new revision has no effect on existing sidecars. They move only when the namespace label points to the new revision and the pods restart, and the istio-injection label takes precedence over istio.io/rev, so a namespace carrying both never moves. The control plane may be one version ahead of the data plane but never behind. Gateways installed with the default profile are upgraded in place to the new revision, and rolling back means reinstalling them for the previous revision by hand.
Environment
One or more Kubernetes clusters running a supported Istio minor in sidecar mode, a default-profile ingress gateway, dozens of namespaces with a mix of istio-injection and istio.io/rev labels, EnvoyFilter customizations and several teams that own their own workloads and restart windows.
Approach
The engagement starts read-only. AceMQ engineers run istioctl x precheck against the cluster, inventory the proxy version in every namespace with istioctl proxy-status and list the labels, webhooks and EnvoyFilters in use. The plan then fixes the order of namespaces, the revision and revision tag layout, the restart sequence for workloads and the checks that must pass before the next namespace moves.
Solution
- Pre-upgrade check with istioctl x precheck and an inventory of proxy versions and control plane revisions per namespace
- Namespaces that carry both istio-injection and istio.io/rev, and the label cleanup needed before they can move
- Revision layout using revision tags such as prod-stable and prod-canary, with istioctl tag list as the record of which namespaces follow which revision
- Restart order for workloads with kubectl rollout restart, and the istioctl proxy-status check that each proxy now follows the new istiod
- Gateway plan: whether to keep in-place upgrades or run revision-specific gateways, and the manual reinstall step a rollback needs
- Review of EnvoyFilters that insert relative to built-in filters or match on proxyVersion, because Istio documents that these can stop applying after an upgrade
- Removal of the old control plane with istioctl uninstall --revision only after the last namespace has moved
Outcome
The engagement delivers a written upgrade plan with the order of namespaces, the revision and tag layout, the restart sequence, the checks between steps and a rollback for each step, plus the precheck output and the proxy version inventory. The platform team's engineers can run the plan, and AceMQ can carry it out alongside them if the team wants.