Back to all use cases

SupportFinancial Services / Payments

Routine node patching kept taking the messaging cluster down

Regional Payments Processor

Stopping a three-member RabbitMQ cluster from losing quorum every time EKS node groups were drained for patching, by fixing readiness, topology spread, disruption budgets and the drain procedure.

Overview

Managed node group upgrades on EKS drain nodes one after another. For a three-member quorum cluster on a StatefulSet, a drain that overlaps with a member still recovering leaves the cluster without a majority. The PodDisruptionBudget that should prevent it is only as good as the readiness signal behind it.

Challenge

Every monthly patch window caused 5 to 15 minutes of publish failures. Two of the three broker pods were sometimes scheduled into the same availability zone, and the readiness probe only checked that the AMQP port was open, so a restarted member was reported ready before it had rejoined the cluster. The drain then moved on to the next node, and engineers recovered the cluster by hand after each window.

Environment

Amazon EKS 1.34 with managed node groups across three availability zones, a three-member RabbitMQ cluster run by the RabbitMQ Cluster Operator, gp3 volumes, and Prometheus alerting.

Approach

AceMQ reproduced the failure on a staging cluster by draining nodes in sequence while watching quorum queue membership, then fixed the three layers that allowed it: scheduling, readiness and the drain procedure. Each change went through the customer's GitOps repository and was tested with a scripted drain before it reached production.

Solution

  1. Reproduced the quorum loss on staging with a scripted rolling drain and recorded member state at every step
  2. Replaced the port-open readiness check with one that confirms the member has rejoined the cluster and its quorum queues are in sync
  3. Added topology spread constraints and pod anti-affinity so each member sits in a different availability zone and on a different node
  4. Set the PodDisruptionBudget to keep two members available and verified that drains respected it
  5. Staggered managed node group updates with a pause between nodes sized to the measured recovery time
  6. Wrote a drain and recovery runbook for the platform team, including the commands that confirm member state before the next node is drained

Outcome

The next three patch windows completed without a publish failure, and the platform team now schedules node updates in working hours instead of waiting for a quiet weekend.

Technologies

KubernetesRabbitMQAmazon EKSPodDisruptionBudgets
  • Consulting

    RabbitMQ Kubernetes Stabilization

    Smart Home Security PlatformCross-Industry

    Hardening RabbitMQ in Kubernetes environments with StatefulSet tuning, quorum queue optimization, storage isolation, and memory/network configuration.

    RabbitMQKubernetes
    Read case study: RabbitMQ Kubernetes Stabilization
  • Consulting

    IBM MQ to RabbitMQ Migration for Insurance Carrier

    American National InsuranceHealthcare

    AceMQ led the phased migration of American National Insurance's IBM MQ infrastructure to RabbitMQ, including legacy code refactoring and HIPAA-compliant data handling across Windows and mainframe queue environments.

    IBM MQRabbitMQKubernetes
    Read case study: IBM MQ to RabbitMQ Migration for Insurance Carrier
  • Consulting

    Replacing Kafka with Debezium + RabbitMQ for Change Data Capture

    American National InsuranceHealthcare

    American National Insurance replaced a Kafka-based CDC pipeline with a standalone Debezium + RabbitMQ architecture, simplifying operations while maintaining SQL Server change capture with improved message routing flexibility.

    KafkaRabbitMQDebeziumKubernetes
    Read case study: Replacing Kafka with Debezium + RabbitMQ for Change Data Capture
  • Support

    Stopping a Pub/Sub Redelivery Spiral on GKE Subscribers

    Digital Wallet ProviderFinancial Services

    Subscribers on GKE fell behind, Pub/Sub redelivered messages they were still working on, and members received duplicate notifications. AceMQ found the gap between the ack deadline and the handler.

    Google Cloud Pub/SubKubernetesJavaPostgreSQL
    Read case study: Stopping a Pub/Sub Redelivery Spiral on GKE Subscribers

Related Reading

Need Expert Kubernetes Support?

AceMQ's senior Kubernetes engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.