Back to all use cases

AssessmentCross-Industry

A control plane reviewed and an upgrade path written down before the next release arrives

Infrastructure Team

A typical engagement for a team running OpenStack on its own hardware: a review of the RabbitMQ message bus and the database behind the control plane, and a written SLURP upgrade plan rehearsed on a copy.

Overview

A typical engagement starts with a private cloud on a release that upstream lists as Unmaintained, deployed with Kolla-Ansible. API calls sometimes time out while every service reports healthy, and the team that built the cloud has shrunk. The team wants the control plane stabilized and a way to reach a maintained release.

Challenge

Every OpenStack service depends on RabbitMQ and on the database. RabbitMQ 4.0 removed classic queue mirroring, oslo.messaging now recommends quorum queues, and Kolla-Ansible documents a queue-durability change in the Epoxy release with a procedure to follow before upgrading. SLURP rules decide which releases can be reached directly, and rolling upgrades between SLURP releases are not guaranteed, so downtime may be part of the plan.

Environment

OpenStack deployed with Kolla-Ansible on the customer's own servers, control nodes running RabbitMQ and a MariaDB Galera cluster, and monitoring that covers hosts but not queue depth or memory alarms.

Approach

AceMQ engineers review RabbitMQ cluster state, queue types, memory alarms and consumer lag, the database cluster and its backup coverage, and the configuration kept in Git against what is deployed. They then write the upgrade path from the current release to a maintained one, naming every release, every RabbitMQ version step and every database step, and rehearse it on a copy of the control plane.

Solution

  1. RabbitMQ review: queue types in use, memory alarm and connection behavior, and the oslo.messaging settings that apply
  2. Database review: Galera cluster state, backup coverage including the default-shard limit of the Kolla-Ansible backup command, and a written restart order
  3. Configuration drift check between Git and the running deployment
  4. Upgrade path to a maintained release using SLURP rules, with the queue-durability procedure and RabbitMQ version steps included
  5. Rehearsal on a copy of the control plane, with the downtime timed and a rollback point for each release
  6. Monitoring additions for queue depth, memory alarms and consumer lag

Outcome

The engagement delivers a written control-plane review with ranked findings, an upgrade plan the customer's operators can run, the rehearsal results and runbooks for the message bus and database. The customer decides when to upgrade, and AceMQ produces no OpenStack patches.

Technologies

OpenStackRabbitMQMariaDBKolla-Ansible

Ready for an OpenStack Health Check?

AceMQ's senior OpenStack engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.