Overview
A telecom billing provider used Google Cloud Pub/Sub between rating, invoicing and dispute-handling services. Each subscription had a dead-letter topic configured in Terraform, yet the same unprocessable usage records were retried for days, and the team had no way to list what had failed.
Challenge
Two separate gaps hid behind a configuration that looked complete. Pub/Sub forwards a message to a dead-letter topic only when its service agent can publish to that topic and acknowledge on the source subscription, and on four of the six subscriptions those roles had never been granted, so nothing was forwarded and delivery attempts were not counted. On the other two, the dead-letter topic had no subscription of its own, so forwarded messages were not retained anywhere anyone could read them.
Environment
Google Cloud Pub/Sub topics and pull subscriptions, consumed by Python workers, with topics, subscriptions and IAM bindings managed in Terraform across development, staging and production projects.
Approach
AceMQ read the dead-letter policy, the IAM bindings and the delivery attempt data for every subscription, then pushed a deliberately bad message through each one in staging and watched where it ended up. The pattern of which subscriptions forwarded and which did not pointed straight to the missing grants.
Solution
- Granted the Pub/Sub service agent the publisher role on each dead-letter topic and the subscriber role on each source subscription, in Terraform
- Created a subscription on every dead-letter topic with a retention period long enough for triage
- Set maximum delivery attempts and an exponential backoff retry policy per subscription, based on how long its downstream dependency takes to recover
- Separated transient failures from poison records in the workers, so a downstream outage retries instead of draining valid usage records into the dead-letter topic
- Added an alert on dead-letter message count and a small tool to inspect a dead-lettered record and republish it to the source topic
- Added a Terraform check so a new subscription cannot be created with a dead-letter policy and without its bindings
Outcome
A poison record now leaves its subscription after the configured attempts and appears in a dead-letter subscription that the team watches and can replay from. The earlier backlog of retried records was traced, classified and cleared.
Technologies
Google Cloud Pub/Sub
Terraform
Python
Kubernetes
dbt
Airflow
Prometheus
