Overview
A records agency kept case files in Tanzu Postgres on its own Kubernetes platform. Backups ran every night and reported success, but nobody had restored one into a clean environment or measured how long it would take. A compliance review prompted an assessment of recoverability.
Challenge
Backup jobs completing is not the same as being able to recover to a point in time. The assessment found that continuous WAL archiving to the object store had been failing since a credential rotation weeks earlier. Nothing alerted, so the point-in-time window had shrunk to the last successful base backup.
Environment
Tanzu Postgres on an on-premises Kubernetes cluster with an S3-compatible object store for backups and WAL archives, in an air-gapped network.
Approach
AceMQ treated recoverability as a measurement. It restored the latest base backup plus archived WAL into a fresh namespace, timed it, and then checked the archive chain, retention and credentials that make a point-in-time recovery possible.
Solution
- Restored the latest base backup and WAL into a clean namespace and recorded the elapsed recovery time
- Read pg_stat_archiver and found the failed archive count and the date the failures began
- Corrected the object store credentials and replayed the missed archive segments where they were still on disk
- Reviewed the retention policy against the agency's required recovery window
- Added an alert on archive failures and a scheduled restore test into a throwaway namespace
- Documented the recovery procedure with the measured times, ready for the compliance review
Outcome
The agency learned its true recovery time and closed the archive gap before it mattered. The restore test now runs on a schedule, and the compliance review received measured figures rather than an estimate.
Technologies
PostgreSQL
Kubernetes
Linux
Prometheus
Terraform
Kong
