Overview
A payments processor runs its ledger on Tanzu Postgres on Kubernetes with two replicas. After a deployment, the primary hit its connection limit and stopped serving queries, yet the pod stayed Ready and no failover happened. Payments timed out for the best part of an hour until an engineer deleted the pod by hand.
Challenge
The readiness probe only confirmed that the port accepted a connection. A primary that has exhausted max_connections still completes a TCP handshake, so the operator saw a healthy leader and never promoted a replica. The same gap would have hidden a full data volume or a primary blocked on a synchronous standby.
Environment
Tanzu Postgres instance with one primary and two replicas on a managed Kubernetes platform, pgbouncer in front, serving a high-volume ledger service.
Approach
AceMQ reproduced the failure in a non-production namespace by exhausting connections and then filling a volume, confirming the probe passed in both cases. The probe and pooling were redesigned together, and each failure was induced again to prove the fix rather than assume it.
Solution
- Reproduced connection exhaustion and a full data volume in staging and recorded what the probe reported
- Replaced the port check with a probe that runs a real query and fails when the database cannot serve it
- Placed pgbouncer in transaction mode in front of the database, sized to max_connections rather than to the application's pool settings
- Set the application pool maximums per pod so a rolling deployment could not exceed the server ceiling
- Induced each failure again after the change and timed the operator's promotion
- Wrote a failover runbook covering the probe, the pooler and manual promotion
Outcome
In every induced failure the operator now promoted a replica within the probe's failure threshold, and deployments no longer pushed the primary to its connection limit. The customer has measured failover times instead of an assumption.
Technologies
PostgreSQL
Kubernetes
Prometheus
Linux
Terraform
Docker
