Overview
A digital wallet provider published transaction events to Google Cloud Pub/Sub for a notification service running on GKE. During busy hours the oldest unacknowledged message age climbed, the pods looked healthy on every dashboard, and members began to receive the same notification more than once.
Challenge
The notification handler called a fraud-scoring service whose latency rose under load, so processing took longer than the subscription's ack deadline. Pub/Sub treated those messages as undelivered and sent them again while the first attempt was still running, which doubled the work and slowed the handler further. The Java subscribers had no flow control, so each pod pulled far more messages than its threads could work and the client could not extend the deadlines in time.
Environment
A Pub/Sub topic and pull subscription consumed by Java subscribers on GKE, with a PostgreSQL database for notification state and an internal fraud-scoring service called in the handler.
Approach
AceMQ ran three live sessions with the customer's engineers. The first compared expired ack deadlines, oldest unacked message age and handler p99 over a busy week. The second reproduced the spiral in staging with a slowed dependency. The third made the changes one at a time, so each could be measured.
Solution
- Matched the subscription ack deadline to the measured p99 of the handler, and left the client library's deadline extension to cover the tail
- Set flow control on the subscriber to a maximum number of outstanding messages and bytes that its worker threads could actually process
- Added a short timeout and a circuit breaker on the fraud-scoring call, so a slow dependency fails a message quickly instead of holding a thread
- Recorded each notification against the message ID in PostgreSQL, so a redelivery is recognised and acknowledged without sending again
- Set an exponential backoff retry policy and a dead-letter topic with a defined maximum delivery attempts
- Alarmed on expired ack deadlines, oldest unacked message age and undelivered message count, with a short runbook for each
Outcome
Backlog during busy hours now drains instead of feeding itself. Duplicate notifications stopped, because a redelivery is recognised before anything is sent, and the on-call engineer sees expired ack deadlines rising before members do.
Technologies
Google Cloud Pub/Sub
Kubernetes
Java
PostgreSQL
RabbitMQ
Docker
Prometheus
OpenTelemetry
Python
