Named senior engineers on call for WAL, replication, and failover events

Overview

A banking platform used logical decoding to feed a change data capture pipeline. When the downstream consumer stalled, the replication slot stopped advancing and the primary retained every WAL segment since the stall. Disk filled twice before AceMQ took over support with named senior engineers and direct escalation, no tier-1 triage.

Challenge

A retained replication slot is one of the few conditions that can take a PostgreSQL primary down through a mechanism unrelated to query load. The customer's monitoring watched disk usage but not slot lag, so the first signal was a disk alert with minutes of headroom left. Distinguishing a temporarily slow consumer from a genuinely abandoned slot required judgment the on-call rotation did not have.

Environment

PostgreSQL on AWS with logical replication feeding a Debezium and Kafka change data capture pipeline.

Approach

AceMQ instrumented slot lag as a first-class signal ahead of disk usage, then established policy for slot lifecycle: what triggers investigation, what triggers a consumer restart, and what justifies dropping a slot. Guardrails were configured so a stalled consumer degrades the pipeline instead of the primary.

Solution

• Added replication slot lag and WAL retention size to monitoring with thresholds well ahead of disk exhaustion
• Configured max_slot_wal_keep_size so a stalled consumer invalidates its slot rather than filling the WAL volume
• Separated WAL onto its own volume with independent capacity and alerting
• Documented slot lifecycle policy covering investigation, consumer restart, and safe slot removal
• Reviewed checkpoint and WAL sizing settings that were driving unnecessary write amplification
• Established 24/7 escalation directly to senior PostgreSQL engineers for slot and replication incidents

Outcome

Technologies