Overview
An enrolment platform processed benefit elections through Amazon SQS queues and Lambda functions. Support kept finding members whose election had never been applied, and the dead-letter queue that should have held the failed event was empty or missing the message. The team believed the queue was working, because its depth alarm had never fired.
Challenge
Three settings worked against each other. Standard queues keep the original enqueue time when a message moves to a dead-letter queue, so a message that spent ten days in the source queue was deleted from a dead-letter queue with a four-day retention within hours of arriving. The redrive policy sent a message to the dead-letter queue after a single receive, so a short downstream outage moved perfectly valid events there. The team's recovery script re-sent dead-lettered messages as new ones, which created duplicates for members whose election had in fact been applied.
Environment
Standard Amazon SQS queues with dead-letter queues, consumed by AWS Lambda functions that wrote to a relational database, with queues and policies managed in Terraform.
Approach
AceMQ pulled the retention and redrive settings for every queue and dead-letter queue pair, compared them with the time failures actually took to be noticed, and replayed one failed event end to end in a test account. The findings were grouped by what they would lose if left alone.
Solution
- Set every dead-letter queue to the maximum 14-day retention, so it always outlives its source queue's retention
- Raised maxReceiveCount from one to five so transient downstream errors retry before a message is dead-lettered
- Alarmed on any message visible in a dead-letter queue, and on age of the oldest message in the source queue
- Replaced the recovery script with the SQS dead-letter queue redrive to the source queue, which preserves the message instead of re-sending a copy
- Made the Lambda writes idempotent on the election ID so a redriven or redelivered message cannot apply twice
- Captured the retention, receive count and alarm settings in the Terraform modules so new queues start with them
Outcome
Failed events now stay in the dead-letter queue long enough to be triaged, an alarm names the queue when one arrives, and redrive no longer produces duplicates. The team could list the members affected by the earlier losses from application logs and correct them.
Technologies
Amazon SQSAWS LambdaAWS
Terraform
Docker
PostgreSQL
Java
Python
