Emergency Incident Response
Broker pods crash-looping, quorum lost, or a cluster that won't reform after a node event. A senior engineer joins a live bridge within 15 minutes with kubectl access to diagnose — not a ticket acknowledgement.
Kubernetes Support
Last reviewed by Scott Sternloff
AceMQ supports production Kubernetes on EKS, AKS, GKE, OpenShift, and on-premises clusters, with particular depth in the stateful workloads that are hardest to run there: message brokers, databases, and caches. StatefulSet pods wedged in Terminating, quorum lost after a node drain, JVMs OOMKilled by cgroup limits, failed cluster upgrades, and storage classes that can't sustain fsync. Every ticket reaches a named senior engineer.
Planning a build, upgrade or migration instead? See AceMQ Kubernetes consulting and services.
Customers




Partners


Incident Triage
These are real symptoms from real Kubernetes production environments — and the first thing our engineers check when one comes in.
| Symptom you're seeing | What we check first | Typical resolution |
|---|---|---|
| Broker StatefulSet pod stuck Terminating, cluster won't reform | Whether the PVC is still bound and which node holds the volume attachment. A node that went NotReady leaves the pod in Terminating and the volume attached, so the replacement pod cannot schedule — and force-deleting the pod before the volume detaches risks two brokers writing one disk. | Under 1 hour |
| Split-brain or lost quorum after a routine node drain | The PodDisruptionBudget against actual quorum requirements. A PDB with maxUnavailable: 1 on a 3-node cluster still permits the drain that takes you to two members while one is mid-restart, so the drain reports success and the cluster loses majority. | 1-3 hours |
| Broker pods OOMKilled with heap usage well under the configured max | Whether the JVM sees the cgroup limit. Without UseContainerSupport and a percentage-based MaxRAMPercentage, the JVM sizes heap from node memory, not the pod limit — and off-heap buffers, metaspace, and page cache push the container past its limit long before heap looks full. | Under 2 hours |
| Publish latency spikes and write stalls with no CPU pressure | The storage class IOPS and throughput ceiling against the broker's fsync rate. gp2 or a default network storage class will throttle at exactly the wrong moment; we correlate volume-level burst-balance metrics with broker flush timings before touching the broker config. | 2-4 hours |
| Nodes join then immediately partition — clustering never stabilizes | NetworkPolicy and service mesh rules on the inter-broker ports. Erlang distribution (4369 and 25672), Kafka's controller and replication listeners, and headless service DNS all get blocked by default-deny policies or silently mangled by an mTLS sidecar that shouldn't be intercepting them. | 2-4 hours |
| Pod restarts under load — liveness probe killing a healthy broker | Probe timeout against broker behavior during GC, compaction, or recovery. A liveness probe with a 1-second timeout will kill a broker replaying its journal after restart, producing a restart loop that looks like a crash but is entirely self-inflicted. | Under 1 hour |
| Broker data lost or unbound after a StatefulSet update | The volumeClaimTemplate, reclaim policy, and whether the StatefulSet was recreated rather than patched. Deleting and reapplying a StatefulSet with a Delete reclaim policy destroys the PVCs — and Helm chart upgrades that change immutable fields quietly encourage exactly that. | Same day |
| Rolling upgrade hangs partway, half the cluster on the new version | The readiness probe against the broker's own cluster-health definition. StatefulSet rolling updates wait for readiness before moving on, so a probe that reports ready before the node has fully rejoined — or never reports ready at all — leaves the rollout mid-flight with mixed versions. | 1-3 hours |
Resolution times reflect typical Kubernetes engagements under an active AceMQ support contract. Every P1 closes with a written root-cause analysis.
What's Included
No add-on pricing for incidents. No per-ticket charges. One contract covers the whole surface.
Broker pods crash-looping, quorum lost, or a cluster that won't reform after a node event. A senior engineer joins a live bridge within 15 minutes with kubectl access to diagnose — not a ticket acknowledgement.
Every P1 closes with a written RCA: what failed, why, the fix applied, and the manifest, probe, or policy change that prevents recurrence. Delivered as standard, not on request.
Storage class selection, IOPS and throughput sizing against broker fsync patterns, volume expansion procedures, and PVC retention policy — reviewed before you discover the limits during an incident.
PodDisruptionBudgets that actually protect quorum, anti-affinity across zones and nodes, topology spread constraints, and drain runbooks your platform team can execute without paging us.
RabbitMQ Cluster Operator, Strimzi, and vendor Helm charts — reconcile failures, CRD upgrades, values drift, and the version-skew problems that surface when the operator and broker upgrade on different schedules.
Moving brokers onto Kubernetes from VMs or bare metal, and moving them back off when the operational cost isn't justified. Both directions, with parallel-run validation and zero message loss.
Why AceMQ
The same senior engineers stay on your account. They know your cluster layout, your storage class, and your broker topology — so a P1 call starts with diagnosis, not twenty minutes of you explaining which cloud and which operator.
You reach a senior engineer directly by phone, email, or Slack. No help desk collecting information to pass along, and no escalation approval process standing between you and someone who can actually fix it.
Your Kubernetes vendor doesn't know RabbitMQ internals. Your broker vendor doesn't debug CNI or CSI. Broker-on-Kubernetes incidents live exactly in that gap, and both parties will tell you the problem is on the other side.
Engineers across 26+ countries and every time zone. Your 3am node-drain incident is someone's mid-afternoon — no overnight skeleton crew, no waiting for a region to wake up.
Quarterly reviews of your manifests, probes, disruption budgets, and storage headroom. When a chart or operator version introduces a regression on one customer's cluster, every affected customer hears about it before their next upgrade window.
We work in environments with no outbound network access, private registries only, and change control that measures deployments in weeks. Support that assumes you can just pull a new image is not support for these clusters.
Cloud, Kubernetes, bare metal, hybrid, and air-gapped — including environments where you can't give us outbound network access.
Escalation Path
Every tier reaches a senior Kubernetes engineer. There is no tier-1 triage layer to get through.
P1 escalation path
A replay of one P1, from the first page to the written root-cause analysis
15-minute P1 response, 24/7.
A response time, not a resolution time.
P1 opened, on-call engineer paged
Phone, email or Slack reaches the on-call senior engineer directly, whether the incident is in RabbitMQ, Kafka, Redis, PostgreSQL or Kubernetes. No web form, no tier-1 queue.
Senior engineer on your bridge
A senior engineer who already knows your environment joins a live bridge, so nobody has to re-explain your topology.
Specialists across the stack join
When an incident crosses components, the engineers who know each one join. For RabbitMQ, the path runs to the core team.
Written root-cause analysis
Root cause, the fix applied and the prevention steps, delivered after every P1.
Pricing
AceMQ Kubernetes support contracts typically run $24,000–72,000 a year, depending on SLA level, scope (number of clusters or nodes, and environments) and deployment (self-managed, cloud-managed or Kubernetes).
| What moves the price | How it changes the contract |
|---|---|
| SLA level | Business-hours cover costs less than 24/7 cover with a 15-minute emergency response. |
| Scope | More clusters or nodes, and more environments, make a larger contract. |
| Deployment | Self-managed, cloud-managed and Kubernetes estates need different amounts of hands-on work. |
Covering several technologies under one contract, for example RabbitMQ, Kafka and Redis, earns an aggressive discount.
Support In Practice
Kubernetes support and stabilization work from AceMQ. Each card links to the full write-up: the cluster, what AceMQ diagnosed or changed, and the result.
Global Aviation Technology Provider
Troubleshooting cluster failover, partition handling, and quorum queue issues in a high-stakes aviation operational environment.
Fortune 500 Software Company
Migration to supported RabbitMQ versions with managed services, standardization, compliance posture, and Tanzu commercial licensing.
Global Telecom Leader
Resolving weekly RabbitMQ crashes, optimizing for 300,000+ connected devices, and architecting horizontal scaling strategy.
Enterprise IT Management Company
Implementing a comprehensive CVE patching and compliance strategy across 10,000+ RabbitMQ deployments running end-of-life versions, with real-time vulnerability monitoring and phased upgrade planning.
Global Payments Processor
Emergency remediation of an Elasticsearch cluster that would not return to green after a data node failure, with replicas blocked by disk watermarks.
SaaS Observability Vendor
Ongoing 24/7 support for an Elasticsearch estate suffering repeated parent circuit breaker trips and long garbage collection pauses under aggregation load.
Support options
AceMQ offers Kubernetes support in four tiers, each handing more of the day-to-day work to AceMQ: 24/7 support and incident management for clusters your team runs, proactive monitoring that finds sizing, probe and upgrade problems first, managed Kubernetes on your own infrastructure, and fully managed Kubernetes on AceMQ infrastructure. EKS, AKS, GKE, OpenShift and self-managed clusters are covered in all four, on the current Kubernetes release and the five end-of-life release lines before it.
You run it, we back you up
Best for: Platform teams that run Kubernetes themselves and need senior cover when a stateful workload breaks at 3am.
You run it, we watch it with you
Best for: Teams that want the rollout that wedges or the drain that loses quorum found in a review, not in production.
We run it in your cloud or data centre
Best for: Regulated estates that must keep clusters, and the data on them, inside their own boundary.
We host it and run it, as a service
Best for: Teams that want brokers, databases and caches on Kubernetes without running a platform team.
| Who does what | Tier 1 24/7 support | Tier 2 Proactive monitoring | Tier 3 Managed, your infrastructure | Tier 4 Fully managed by AceMQ |
|---|---|---|---|---|
| Who runs Kubernetes day to day? | Your team | Your team, with AceMQ reviews | AceMQ | AceMQ |
| Where does it run? | Your infrastructure | Your infrastructure | Your cloud account or data centre | AceMQ infrastructure |
| 24/7 incident response | 15-minute P1 response | 15-minute P1 response | 15-minute P1 response | 15-minute P1 response |
| Health checks and CVE alerts | On request | Scheduled | Included | Included |
| Patching and upgrades | Your team, AceMQ advises | Planned with AceMQ | Done by AceMQ | Done by AceMQ |
| Monitoring and on-call | Your team | Your team | AceMQ | AceMQ |
| Data, keys and accounts | Yours | Yours | Yours | Your data, hosted by AceMQ |
Every Tier can start as an ad hoc engagement or an annual partnership, and you can move up or down a Tier as your team changes.
One partner for Kubernetes
Support is one part of what AceMQ does for Kubernetes. The same engineers deliver consulting, health checks, migrations and end-of-life cover, and support the systems that usually run next to it.
We support production Kubernetes clusters and the workloads on them, with the deepest experience in stateful systems such as RabbitMQ, Tanzu RabbitMQ, Kafka, ActiveMQ, databases, and caches. That covers the full stack that keeps them running: StatefulSets, PVCs and storage classes, probes, PodDisruptionBudgets, network policy, operators, and Helm charts. Coverage includes 24/7 emergency response with a 15-minute SLA, root cause analysis on every P1, upgrade and migration support, and quarterly manifest reviews. Stateful workloads are where Kubernetes breaks most often, and that is where our engineers spend most of their time.
Yes, and it is our single largest Kubernetes workload. We are the number one RabbitMQ partner globally with direct access to the RabbitMQ core team, and we support both the RabbitMQ Cluster Operator and hand-rolled StatefulSet deployments. That covers quorum queue membership after node replacement, Erlang distribution over CNI, cert-manager TLS rotation breaking inter-node auth, and the operator reconcile failures that leave a cluster mid-upgrade.
Yes. We support Kafka on Kubernetes including Strimzi-managed clusters, covering broker and controller pod scheduling, storage sizing for log segments and fsync patterns, rack awareness mapped to zone topology, rolling upgrades under partition-availability constraints, and the KRaft migration path. As with RabbitMQ, most incidents turn out to be a platform-layer cause with a broker-layer symptom.
Because heap is not the whole story inside a container. A JVM without container awareness sizes its heap from node memory rather than the pod limit, and even a correctly sized heap sits alongside metaspace, thread stacks, direct byte buffers, and native memory used by the broker's network layer. The container limit counts all of it. The fix is usually MaxRAMPercentage plus explicit direct-memory limits, sized against measured off-heap usage rather than a guess.
P1 emergencies — brokers down, quorum lost, messages not flowing — get a 15-minute first response with a senior engineer joining a live bridge. P2 critical issues are 1 hour, P3 high-priority 4 hours, and P4 standard questions next business day. These are contractual, not best-effort targets.
Alongside them, on the stateful workload. Platform teams are generally strong on the cluster and weaker on what a broker needs from it — the fsync behavior that makes a storage class inadequate, the drain sequence that costs you quorum, the probe timeout that kills a recovering node. We work directly with your platform team rather than around them, and most of our recommendations end up in their manifests, not ours.
Kubernetes is the open source container orchestration platform from the Cloud Native Computing Foundation (CNCF). The Kubernetes community maintains the three most recent minor releases, each for about a year, and the Kubernetes project answers questions through documentation, Slack and GitHub with no response time. Managed Kubernetes services, Amazon EKS, Azure AKS and Google GKE, support the control plane, the API server, etcd and the scheduler, under your cloud support plan, with their own version calendars. Distribution vendors such as Red Hat OpenShift support their platform. None of them supports the workloads you deploy: your Helm charts, operators, ingress, container images, and how your pods use compute, storage and the network. AceMQ covers that layer, and the whole Kubernetes cluster for self-managed deployments on virtual machines or bare metal, so a stateful workload that loses quorum during a node drain has someone accountable for it.
Everything that keeps a Kubernetes cluster and its workloads running: cluster upgrades one minor version at a time with deprecated API scans of every Helm release and operator first, node pools and autoscaling, the container runtime and registry, CoreDNS and NetworkPolicy, ingress controllers and service meshes, persistent volumes and storage classes, role-based access control and secrets, and etcd health on self-managed control planes. Stateful workloads are where we have the deepest experience: message brokers, databases, caches and search clusters running as StatefulSets, which fail in ways that cloud native tooling was not designed for. Small platform teams use us to deploy and manage Kubernetes with 24/7 cover they cannot staff themselves.
Yes. AceMQ works on Kubernetes in two ways: ad hoc engagements for a defined piece of work, such as a health check, an upgrade or an incident, or an annual partner engagement that covers 24/7 support, proactive reviews and named engineers. Many teams start ad hoc and move to an annual partnership once the first engagement is done.
Every AceMQ proposal is tailored to your requirements: your Kubernetes environment, the SLA you need, the hours of cover and the scope of work. Most contracts run $24,000–72,000 a year, depending on the SLA, the scope and how Kubernetes is deployed, with aggressive discounts when several technologies share one contract. Request a quote and AceMQ replies within 1 hour.
Let's talk
Reactive cover, proactive reviews, or AceMQ running Kubernetes for you on your infrastructure or ours. A senior engineer scopes it with you, and quote requests are answered within 1 hour.
Production incident right now?Email a Kubernetes Architect
Send us a message and we'll follow up within 1 hour — or book a free 30-min consultation directly.
305-204-2607info@acemq.comPrefer to talk now? Call us directly or use the consultation tab to find a time that works.