Overview
An analytics vendor saw write latency climb every afternoon on its Tanzu Postgres cluster and recover by evening. The team had tuned checkpoint settings twice with no lasting effect. AceMQ took the problem on under a support contract and treated the volume as a suspect from the start.
Challenge
The data and WAL shared one volume type that grants burst I/O credits, and the afternoon load drained them. Once the credits ran out, checkpoints and WAL flushes slowed sharply, which showed up as latency in the application. PostgreSQL settings could soften it, but they could not create IOPS the volume no longer had.
Environment
Tanzu Postgres on Amazon EKS, one primary and two replicas, general-purpose network volumes for data and WAL, Prometheus and Grafana for metrics.
Approach
AceMQ lined up volume credit balance, checkpoint duration and write latency on one timeline across several days. The correlation was exact, so the work moved to the storage class first and to checkpoint behaviour second.
Solution
- Plotted volume burst balance against checkpoint write time and p99 commit latency over a week
- Moved WAL to its own volume with provisioned IOPS and sized the data volume to the sustained load
- Raised max_wal_size and set checkpoint_completion_target so checkpoints spread their writes over the interval
- Reviewed bgwriter behaviour and shared_buffers against the working set
- Applied the change replica first, then switched over, using the operator's rolling procedure
- Added alerts on volume credit balance and checkpoint duration to the on-call rota
Outcome
The afternoon spikes stopped and p99 commit latency stayed flat through the daily peak. The customer now has a volume-credit alert that would have flagged the cause on the first day.
Technologies
PostgreSQL
Kubernetes
Prometheus
Terraform
Linux
Kafka
