Overview
This is a typical engagement for a platform team that runs several self-hosted Argo CD instances, one per environment or business unit, which have drifted onto different minor versions. The project patches only its three newest minor versions and ships a new minor roughly every three months, so one of those instances is always about to fall out of the window. AceMQ and a partner GitOps specialist assess the fleet and plan the upgrades.
Challenge
Each minor has its own upgrade notes with breaking changes, and an instance several minors behind has to read every set between its version and the target. At the same time the largest instances, which manage the most clusters, may be running one application controller replica that is already short of memory, so an upgrade that restarts it needs capacity planned first.
Environment
Several Argo CD instances installed from the upstream manifests on Kubernetes, tens of registered clusters between them, ApplicationSets that generate most of the applications, Redis in HA mode on the larger instances, and Git repositories shared across teams.
Approach
AceMQ lists every instance with its minor version, install method, cluster and application counts and controller settings, then reads the upgrade notes for every minor hop. Findings come first and no instance is changed during the assessment.
Solution
- Lists every Argo CD instance with its minor version and its position in the three-minor patch window
- Reads the upgrade notes for each minor between the current version and the target and records the breaking changes that touch the settings in use
- Reviews controller replicas, the sharding method, repo-server capacity and Redis against the number of clusters and applications
- Plans a rehearsal on a non-production instance using a copy of real Applications and ApplicationSets
- Orders the upgrades by risk, with a settings export and a rollback boundary for each instance
- Writes the calendar that keeps every instance inside the window as new minors ship
Outcome
The engagement delivers an instance-by-instance table of versions and window position, the list of breaking changes that apply to the settings in use, a controller and repo-server sizing review, a rehearsal plan, a dated upgrade order with rollback boundaries and a calendar for the next minors. The platform team gets a plan it can schedule instead of upgrading whichever instance breaks first.
Technologies
Kubernetes
OpenShift
RabbitMQ
Kafka
Kong
