Overview
An energy distributor ran four Tanzu Postgres clusters for metering and billing on an operator release that no longer supported the PostgreSQL version it needed to move to. The upgrade had been deferred twice because nobody could say what order the pieces had to move in.
Challenge
The operator and the database engine each have their own compatibility range, so the operator had to move first, in steps, before the PostgreSQL major version could change. Each step touched production data services with strict availability needs, and the team had no tested way back if a step misbehaved.
Environment
Four Tanzu Postgres clusters on a self-managed Kubernetes platform, each with two replicas, provisioned and configured through Terraform.
Approach
AceMQ mapped the compatibility path first, then rehearsed every step on clones of production data restored into a separate namespace. The major-version upgrade used pg_upgrade on a replica-first basis, with logical replication held as the fallback for the largest cluster.
Solution
- Mapped the supported operator and PostgreSQL combinations and wrote down the order of each step
- Restored clones of all four clusters and rehearsed the operator upgrade, then the major-version upgrade, timing each
- Ran pg_upgrade with link mode on a replica first, and checked extensions and ANALYZE after the move
- Prepared a logical replication path for the largest cluster in case the downtime window proved too short
- Defined a rollback point at every step, backed by a fresh backup that had been restored once
- Updated the Terraform modules so the upgraded versions are the declared state
Outcome
All four clusters moved to the new operator and PostgreSQL major version in planned windows, with the rehearsed timings holding in production. The customer has a documented upgrade path for the next operator release.
Technologies
PostgreSQL
Kubernetes
Terraform
Prometheus
Linux
Docker
Apache Spark
