Overview
A typical engagement starts with a private cloud on a release that upstream lists as Unmaintained, deployed with Kolla-Ansible. API calls sometimes time out while every service reports healthy, and the team that built the cloud has shrunk. The team wants the control plane stabilized and a way to reach a maintained release.
Challenge
Every OpenStack service depends on RabbitMQ and on the database. RabbitMQ 4.0 removed classic queue mirroring, oslo.messaging now recommends quorum queues, and Kolla-Ansible documents a queue-durability change in the Epoxy release with a procedure to follow before upgrading. SLURP rules decide which releases can be reached directly, and rolling upgrades between SLURP releases are not guaranteed, so downtime may be part of the plan.
Environment
OpenStack deployed with Kolla-Ansible on the customer's own servers, control nodes running RabbitMQ and a MariaDB Galera cluster, and monitoring that covers hosts but not queue depth or memory alarms.
Approach
AceMQ engineers review RabbitMQ cluster state, queue types, memory alarms and consumer lag, the database cluster and its backup coverage, and the configuration kept in Git against what is deployed. They then write the upgrade path from the current release to a maintained one, naming every release, every RabbitMQ version step and every database step, and rehearse it on a copy of the control plane.
Solution
- RabbitMQ review: queue types in use, memory alarm and connection behavior, and the oslo.messaging settings that apply
- Database review: Galera cluster state, backup coverage including the default-shard limit of the Kolla-Ansible backup command, and a written restart order
- Configuration drift check between Git and the running deployment
- Upgrade path to a maintained release using SLURP rules, with the queue-durability procedure and RabbitMQ version steps included
- Rehearsal on a copy of the control plane, with the downtime timed and a rollback point for each release
- Monitoring additions for queue depth, memory alarms and consumer lag
Outcome
The engagement delivers a written control-plane review with ranked findings, an upgrade plan the customer's operators can run, the rehearsal results and runbooks for the message bus and database. The customer decides when to upgrade, and AceMQ produces no OpenStack patches.