Ensuring messaging reliability for airport gate management systems worldwide

Overview

The company provides mission-critical airport gate management systems used worldwide. Their RabbitMQ deployment on Kubernetes was experiencing publish failures, cluster failover issues, and quorum queue synchronization problems that threatened system availability.

Challenge

RabbitMQ was co-located with PostgreSQL on shared Kubernetes nodes, causing resource contention. Quorum queue synchronization was failing at 3,000 IOPS storage, and cluster partitions were causing zombie bindings and message publishing failures. The team needed to upgrade to RabbitMQ 4.2 while maintaining stability across Azure and VMware environments.

Environment

Kubernetes clusters across Azure and VMware, RabbitMQ with PostgreSQL co-location, Prometheus/Grafana monitoring, C and Node.js client applications.

Approach

AceMQ conducted a multi-week intensive assessment with daily sessions, performing disk benchmarking with FIO, analyzing RAFT consensus logs, and designing infrastructure isolation strategies. The team provided comprehensive documentation covering failover procedures, monitoring, backup/restore, and federation.

Solution

• Separated RabbitMQ from PostgreSQL onto dedicated nodes with isolated SSD storage
• Recommended premium SSD V2 disks to achieve sub-millisecond latency required for quorum queues
• Developed automated detection and recovery scripts for stuck quorum queues and zombie bindings
• Configured Prometheus metrics and Grafana alerting for proactive cluster health monitoring
• Delivered comprehensive runbook for failover, backup/restore, and federation operations
• Guided upgrade path from RabbitMQ 3.x to 4.2 with stability validation

Outcome

Technologies