Failover that has been tested, not just configured

Overview

A telecommunications operator ran PostgreSQL with manual failover documented in a wiki page that had never been exercised under real conditions. Recovery depended on a small number of engineers being reachable. AceMQ designed and validated an automated high availability architecture to replace it.

Challenge

Automated failover is straightforward to configure and difficult to make correct. The design had to prevent split-brain during network partitions, define which transactions could be lost and which could not, and route application traffic to a new primary without every client needing a restart. Existing replicas were asynchronous with no policy on acceptable data loss.

Environment

PostgreSQL on Kubernetes across three availability zones, backing subscriber provisioning and billing services.

Approach

AceMQ started from the customer's actual recovery point and recovery time requirements per workload rather than applying a uniform posture. Synchronous replication was applied only where the durability requirement justified its latency cost, and the design was proven through deliberate failure injection before handover.

Solution

• Defined recovery point and recovery time objectives per workload and mapped them to concrete replication settings
• Deployed Patroni with a distributed consensus store for leader election and fencing against split-brain
• Applied synchronous replication selectively where durability requirements justified the latency cost
• Designed connection routing so application clients follow the primary without coordinated restarts
• Ran deliberate failure injection covering primary loss, zone loss, and network partition to prove behavior
• Produced runbooks for both automated failover verification and manual intervention when automation is bypassed

Outcome

Technologies