Back to all use cases

ConsultingCross-Industry

A new control plane beside the old one, and a namespace order with a way back

Platform Engineering Team

A typical engagement for a platform team that runs Istio sidecars across many namespaces and needs a canary upgrade plan: a pre-upgrade check, a new control plane revision beside the old one, namespace-by-namespace moves and a rollback written before the first change.

Istio support

Overview

This is a typical engagement for a platform team whose mesh runs an Istio minor that the project is about to stop supporting. Istio supports a minor release until 6 weeks after the minor two releases newer ships, so the team needs an upgrade it can schedule. AceMQ engineers and a partner service mesh specialist plan a revision-based canary upgrade, which Istio documents as safer than an in-place upgrade and recommends.

Challenge

Installing a new revision has no effect on existing sidecars. They move only when the namespace label points to the new revision and the pods restart, and the istio-injection label takes precedence over istio.io/rev, so a namespace carrying both never moves. The control plane may be one version ahead of the data plane but never behind. Gateways installed with the default profile are upgraded in place to the new revision, and rolling back means reinstalling them for the previous revision by hand.

Environment

One or more Kubernetes clusters running a supported Istio minor in sidecar mode, a default-profile ingress gateway, dozens of namespaces with a mix of istio-injection and istio.io/rev labels, EnvoyFilter customizations and several teams that own their own workloads and restart windows.

Approach

The engagement starts read-only. AceMQ engineers run istioctl x precheck against the cluster, inventory the proxy version in every namespace with istioctl proxy-status and list the labels, webhooks and EnvoyFilters in use. The plan then fixes the order of namespaces, the revision and revision tag layout, the restart sequence for workloads and the checks that must pass before the next namespace moves.

Solution

  1. Pre-upgrade check with istioctl x precheck and an inventory of proxy versions and control plane revisions per namespace
  2. Namespaces that carry both istio-injection and istio.io/rev, and the label cleanup needed before they can move
  3. Revision layout using revision tags such as prod-stable and prod-canary, with istioctl tag list as the record of which namespaces follow which revision
  4. Restart order for workloads with kubectl rollout restart, and the istioctl proxy-status check that each proxy now follows the new istiod
  5. Gateway plan: whether to keep in-place upgrades or run revision-specific gateways, and the manual reinstall step a rollback needs
  6. Review of EnvoyFilters that insert relative to built-in filters or match on proxyVersion, because Istio documents that these can stop applying after an upgrade
  7. Removal of the old control plane with istioctl uninstall --revision only after the last namespace has moved

Outcome

The engagement delivers a written upgrade plan with the order of namespaces, the revision and tag layout, the restart sequence, the checks between steps and a rollback for each step, plus the precheck output and the proxy version inventory. The platform team's engineers can run the plan, and AceMQ can carry it out alongside them if the team wants.

Technologies

IstioKubernetesEnvoy

Need Istio Architecture Guidance?

AceMQ's senior Istio engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.