01
HA Review
The entry offer: topology, replication, headroom, backups and monitoring read against how the platform is really used. One to four weeks.
High Availability and Disaster Recovery
Managed Disaster Recovery: High Availability for Brokers and Databases
Last reviewed by Tyler Eastridge
Managed disaster recovery from AceMQ is a project followed by cover: we design high availability and DR for your RabbitMQ, Kafka, Redis or Valkey, and PostgreSQL platforms, prove it with failover tests, write the runbooks, and then support or run the platform. It starts with an HA review that takes one to four weeks, depending on complexity and what we find. You get a written report, a ranked fix list and a tested recovery plan.
- weeks for the HA review, depending on complexity
- 1–4
- min fastest response target, by support plan
- 15
- tiers of operational cover after the project
- 4
Is This You?
You probably need this if…
- 01
A public-sector platform whose node disk filled and took it down
- 02
RabbitMQ split brain keeps happening and nobody is sure which node to trust
- 03
Failover is a wiki page that has never been run under real conditions
- 04
A disaster recovery site exists on paper, and nobody has switched over to it
- 05
An auditor or a risk questionnaire is asking for recovery time and recovery point evidence
- 06
Kafka replication and acknowledgement settings were chosen at go-live and never checked against the durability the business assumes
- 07
One or two engineers know how to recover the platform, and they are not always reachable
Outcomes
Where you are now, and where you end up
Concrete state changes, not deliverable counts. This is what actually differs about your Infrastructure estate when the engagement closes.
Before
High availability is assumed because there are three nodes.
After
Recovery time and recovery point targets set per workload, mapped to concrete replication settings, and checked against what the platform does today.
Before
Failover has been configured but never exercised.
After
Deliberate failure tests for node loss, disk full, zone loss and network partition, with the results written down.
Before
Recovery depends on whoever remembers the steps.
After
Runbooks your own engineers can execute without waiting on a vendor, for the automated path and the manual one.
Before
After the project, nobody owns the platform at 3 a.m.
After
Support or managed operation on the four-tier ladder, from the engineers who designed the recovery.
Scope
What's covered
- 02
RabbitMQ and Kafka
Quorum queue and stream design, replication factor, in-sync replicas and acknowledgements, cross-data-centre replication, and partition behaviour.
- 03
Redis, Valkey and PostgreSQL
Sentinel or Cluster for Redis and Valkey, and streaming replication with Patroni-style leader election for PostgreSQL, with data-loss policy stated per workload.
- 04
Kubernetes and Cloud
Kubernetes high availability through zone spread, anti-affinity, disruption budgets and storage that survives node loss, for brokers and databases run on Kubernetes or in a cloud account.
- 05
Failover Tests
Planned failure injection on a rehearsal environment or inside an agreed window, with a record of what happened and how long it took.
- 06
Runbooks and Handover
Step-by-step recovery for the automated path and the manual path, written for the engineers who will be on the pager.
The Engagement
How it actually runs
Every phase has a defined duration and a concrete artifact handed over at the end of it. You always know what stage you're in and what you've received.
- Phase 11–4 weeks
HA Review
A senior engineer reviews each platform for single points of failure, replication and durability settings, resource headroom, backup and restore, and the monitoring that should have caught the last incident. The 1 to 4 weeks depend on complexity, focus areas and problems found.
You receive
- Written findings with evidence
- Ranked fix list
- Recovery time and recovery point gaps per workload
- Phase 2Scoped per platform
Design
Target topology across failure domains, replication and failover settings, and the cross-site design if disaster recovery spans two data centres or regions.
You receive
- Reference design per platform
- Mapping from each recovery target to its settings
- Change plan for your change windows
- Phase 3Scoped per platform
Failover Tests
We exercise the design: node loss, disk full, zone loss and network partition, measured against the targets. Gaps found are fixed and re-tested.
You receive
- Test plan and results
- Measured recovery times
- Fixes verified
- Phase 4Scoped per engagement
Runbooks and Cover
Runbooks for your team, then the support tier you choose, from on-call escalation to AceMQ operating the platform.
You receive
- Runbooks for automated and manual recovery
- Agreed support tier and escalation path
- Handover session with your engineers
High availability and disaster recovery across your platforms
High availability keeps a platform running through the loss of a node. Disaster recovery brings it back after the loss of a site. The first is designed into the cluster, and the second is designed around it and proven by switching over. One engagement covers both, across the platforms you run.
| Platform | High availability mechanism | What we test | Where to start |
|---|---|---|---|
| RabbitMQ | Quorum queues and streams, replicated by Raft across an odd number of nodes in separate failure domains. Federation or shovel between clusters for disaster recovery. | Node loss, partition behaviour, disk and memory alarms, and queue member counts after node replacement. | RabbitMQ health check |
| Kafka | Replication factor, in-sync replicas and producer acknowledgements set as a group, rack or zone awareness, and a KRaft controller quorum. | Broker loss, ISR shrinkage, controller failover and log-directory headroom. | Kafka health check |
| Redis and Valkey | Sentinel for a primary and replicas, or Cluster for sharding. Replication is asynchronous, so the write-loss window is stated. | Primary loss, failover time, and client behaviour during promotion. | Redis support |
| PostgreSQL | Streaming replication with automated leader election and fencing, for example Patroni with a consensus store. Synchronous replication only where durability justifies the latency. | Primary loss, zone loss, network partition and client routing to the new primary. | PostgreSQL support |
| MySQL | Replicas with automated failover, or a group replication cluster, depending on the version and the durability needed. | Primary loss, replica lag at failover and application reconnection. | MySQL support |
| Kubernetes | Zone spread with anti-affinity, pod disruption budgets, storage that survives node loss and operators that upgrade one member at a time. | Node drain, zone loss and rolling upgrades under load. | Containers and Kubernetes |
Failover, high availability and disaster recovery: the difference
| Term | What it means | How it is judged |
|---|---|---|
| High availability | Designing the platform so that losing a component does not stop service. | Does the service stay up when one node fails? |
| Failover | The act of switching to a standby, automatically or by hand. | How long does the switch take, and what data is lost? |
| Disaster recovery | Restoring service after losing a whole site, region or the data itself. | Recovery time objective and recovery point objective, proven by a drill. |
In short, high availability is the design, failover is the mechanism and disaster recovery is the plan for when the design is not enough. A platform can be highly available and still have no disaster recovery.
Disaster recovery by platform: what sets the recovery point and the recovery time
Two numbers define a disaster recovery design, and your business sets both. The recovery point objective is the maximum acceptable time since the last recovery point, which is how much data you can lose. The recovery time objective is the maximum acceptable delay between the interruption and the restoration of service. The AWS disaster recovery whitepaper uses these definitions, and notes that even a well built strategy has a recovery time and recovery point greater than zero.
Each platform has its own documented recovery primitive, and each has a limit that decides which number you can really meet. AceMQ does not quote a figure in advance. We map your targets to these mechanisms, and a failover test in an agreed window measures what the design reaches.
| Platform | Documented DR mechanism | What limits the recovery point | What limits the recovery time |
|---|---|---|---|
| Kafka | One cluster per data centre, with MirrorMaker 2 copying topics, configs, consumer group offsets and ACLs between them. Confluent Cluster Linking is the managed equivalent on Confluent clusters. | Records not yet mirrored when the site is lost. Each site stands alone if the link is down, and mirroring catches up afterwards. | Pointing clients at the surviving cluster and translating consumer offsets. |
| RabbitMQ | Federation or shovel between two clusters. Tanzu RabbitMQ, the commercial edition, adds Warm Standby Replication, which copies schema definitions and messages from an upstream cluster to a downstream standby. | Federation is asynchronous, and neither plugin replicates acknowledgements, so after a failover messages that were already consumed can be delivered again. | Passive queues keep growing unless you set TTL or length limits, and consumers must tolerate duplicates. |
| PostgreSQL | Continuous archiving of the write-ahead log with point-in-time recovery, plus streaming replication to a standby. | Streaming replication is asynchronous by default, so transactions not yet shipped can be lost. Synchronous commit closes the gap and adds commit latency. | Restoring a base backup and replaying the log, or promoting a standby and re-pointing clients. |
| Redis and Valkey | Primary and replica replication, with persistence enabled on both. | Replication is asynchronous by default, and acknowledged writes can be lost on failover. WAIT narrows the window but does not make either system strongly consistent. | Sentinel or cluster failover time, then client reconnection. |
| Cassandra | Multi-data-centre replication, with NetworkTopologyStrategy setting a replication factor for each data centre. | The consistency level each write uses, such as LOCAL_QUORUM or EACH_QUORUM, decides how many replicas in which data centres acknowledged it. | Routing clients to a surviving data centre. |
How often to test is a decision tied to those two numbers. A platform whose recovery time target is minutes needs rehearsals far more often than one with a day to spare, and any change to topology, version or client libraries is a reason to rehearse again. The AWS whitepaper makes the same point: test your strategy regularly.
Split brain, full disk and failed failover: what we test first
Most outages in this area come from three failures. Each is cheap to rehearse and expensive to meet for the first time in production.
Split brain and network partitions
RabbitMQ high availability depends on a majority of replicas being online. On RabbitMQ 4.3 and later, the metadata store, quorum queues and streams are all Raft-based, the old partition handling strategies are gone, and a three-node cluster tolerates one node failure while a two-node cluster tolerates none. On earlier versions, the partition handling mode and the network settings decide what happens, and those are the settings we review first. For PostgreSQL the equivalent is fencing the old primary through leader election, and for Redis and Valkey it is the Sentinel quorum.
A full disk
A full disk can take down any stateful platform. On RabbitMQ, when free disk space drops below the configured limit an alarm blocks all producers, the alarm is cluster-wide, and if messages are paged out fast enough the disk can still fill between checks and crash the node. On PostgreSQL, an inactive replication slot retains write-ahead log until the volume fills. On Kafka, retention and log-directory headroom decide how long a broker rebuild takes. We size headroom, set alarms that page before the limit, and test what happens at the limit.
A failover that does not work
Two published examples show the shape. A telecommunications operator ran PostgreSQL with manual failover documented in a wiki page that had never been exercised. A government systems integrator had a RabbitMQ failover that was a manual promotion nobody had practised. In both cases the fix was the same: set the recovery targets, automate or write down the path, and prove it with deliberate failure tests before handover.
The HA review: the entry offer
The HA review is where most engagements start. It is an engineer-led read of the platform against how it is used, with written findings and a ranked fix list. Assessments take one to four weeks depending on complexity, focus areas and problems found. We need visibility, not hands-on access: configuration exports and metrics history are enough, and read-only access speeds it up.
- Topology — Node count and placement across failure domains, and single points of failure.
- Replication and durability — Queue types, replication factor, in-sync replicas, acknowledgements and the data-loss window each setting implies.
- Headroom — Disk, memory and file-descriptor limits against sustained and peak load, and the alarms on them.
- Backup and restore — Whether a restore has ever been run, and how long it took.
- Monitoring — What is measured, what pages, and what would have caught the last incident.
Managed disaster recovery: from project to cover
The project ends with a platform that has been designed and tested. The cover after it is your choice, on four tiers.
| Tier | What it is | The question it answers |
|---|---|---|
| Tier 1: Reactive support | Senior engineers on call for incidents under a contracted SLA, with response targets from 15 minutes, by plan. | Our team runs the platform. Who do we call when it breaks at night? |
| Tier 2: Proactive support | Tier 1 plus monitoring through AceMQ's observability setup. | Who notices trouble before our customers do? |
| Tier 3: Managed on your infrastructure | AceMQ operates the platform inside your cloud account or data center. | Data must stay inside our boundary and we do not want the pager. |
| Tier 4: Managed on AceMQ's infrastructure | AceMQ hosts and operates the platform. | We want it run for us, and our policy allows a hosted service. |
What this page does not claim
- No availability figures in advance — We do not promise an uptime percentage before seeing the platform, its workload and its failure domains.
- Tests need your agreement — Failure injection is run on a rehearsal environment or inside a window you approve.
- Application code stays yours — We design, review and test with your engineers. Your developers write the retry and idempotency logic that recovery depends on.
- AceMQ pricing is by quote — We do not publish a price list. Tell us what you run and we come back with a written quote.
FAQ
Questions about Infrastructure high availability and disaster recovery
What is managed disaster recovery?
It is disaster recovery that someone else owns after the design is done. AceMQ designs the high availability and recovery setup for your brokers and databases, tests it with deliberate failures, writes the runbooks, and then supports or operates the platform on one of four tiers, so the recovery plan stays current and rehearsed.
Does PostgreSQL support high availability?
It supports the building blocks: streaming replication, hot standbys and logical replication. It does not detect a failed primary and promote a standby by itself. Automatic failover comes from tooling such as Patroni with a consensus store, or from a managed service, and it has to be designed so that two nodes never both accept writes.
Is Redis Sentinel deprecated?
No. Sentinel is still the documented Redis high availability mechanism for non-clustered deployments, and Valkey has its own Sentinel with the same role. Only Sentinel version 1, from Redis 2.6, is deprecated. Replication is asynchronous, so we state the write-loss window and test failover timing for your clients.
Do you design MySQL high availability too?
Yes. MySQL high availability is designed with replicas and automated failover, or a Group Replication or InnoDB Cluster setup, depending on the version and the durability the workload needs. We test primary loss, replica lag at failover and application reconnection, and MySQL 5.7, 8.0 and 8.4 LTS are covered by our MySQL support.
What is the difference between failover and high availability?
High availability is the design that keeps a service running when a component fails. Failover is the act of switching to a standby, automatically or by hand. Disaster recovery is the plan for losing a whole site. A platform can have failover configured and still not be highly available if the switch has never been tested.
RabbitMQ split brain keeps happening: who can fix our cluster?
AceMQ's RabbitMQ engineers. We read the cluster logs and settings, find what is causing the partitions, and fix the node count, network settings and partition handling for your version. On RabbitMQ 4.3 and later, quorum queues and the metadata store are Raft-based and need a majority online, so the design focuses on failure domains and network reliability.
A full disk took our broker down: who designs DR for RabbitMQ across two data centres?
AceMQ does. We size disk and memory headroom, set the free-disk limit and alarms, and design the cross-site layer, usually federation or shovel between two clusters rather than one cluster stretched across a WAN, with the failover and failback steps tested. On RabbitMQ a disk alarm blocks all producers cluster-wide, so headroom is the first fix.
Who runs failover tests for Kafka and PostgreSQL?
AceMQ senior engineers, alongside your team. We plan the test, run it on a rehearsal environment or inside an agreed window, and record what failed over, how long it took and what data was at risk. Kafka tests cover broker loss, in-sync replica behaviour and controller failover. PostgreSQL tests cover primary loss, zone loss and network partition.
What RPO and RTO can Kafka, RabbitMQ and PostgreSQL meet?
You set the targets, and the design is built to meet them. What each platform can reach depends on its mechanism: mirrored clusters for Kafka, federation or shovel for RabbitMQ, and asynchronous or synchronous replication for PostgreSQL. Every option has a recovery point and time above zero, and the failover test measures what yours really are. We do not quote figures before that.
How often should we test a failover?
There is no universal interval. Test often enough that the result still describes the platform you run: a short recovery time target needs frequent rehearsals, and any change to topology, version or client libraries needs another. Record what failed over, how long it took and what data was at risk each time.
How long does an HA review take?
One to four weeks, depending on complexity, the number of platforms, the focus areas and the problems found. You get written findings with evidence and a ranked fix list. We need configuration exports and metrics history, and read-only access speeds it up.
How much does this cost?
It is scoped on the platforms, the failure domains and the depth of testing, and we do not publish a price list. Tell us what you run and we will reply to your quote request within 1 hour.
Let's talk
Talk to an Engineer About High Availability and Disaster Recovery
Tell us which brokers and databases you run, where they sit and what has gone wrong. We will scope an HA review and reply to your quote request within 1 hour.
Get in Touch
Talk to an AceMQ Expert
Send us a message and we'll follow up within 1 hour — or book a free 30-min consultation directly.
305-204-2607info@acemq.com66 W. Flagler St. 9th Floor
Miami, FL 33130
Miami, FL 33130
Prefer to talk now? Call us directly or use the consultation tab to find a time that works.