Overview
When an SQS queue's visibility timeout is shorter than the consuming Lambda's timeout, a message becomes visible again while the first invocation is still working on it. A second invocation picks it up, and under sustained load this becomes a self-feeding loop of duplicate work that costs real money and produces duplicate side effects.
Challenge
A queue that normally ran near empty accumulated a large backlog overnight, invocation counts were far higher than message counts, and downstream records showed duplicates. The function timeout had been raised to handle larger payloads without a corresponding change to the queue's visibility timeout. Compounding it, a handler error path returned success, so failed messages were deleted rather than sent to the dead-letter queue, and a separate class of poison messages recirculated indefinitely.
Environment
AWS Lambda functions consuming SQS queues, writing to a relational database and calling internal APIs. AceMQ works on the customer's functions, event source configuration, and integration design — not on the AWS platform itself.
Approach
AceMQ compared invocation counts against message counts to confirm duplication, then audited every function-to-queue pairing for timeout relationships, batch behavior, and error handling. The fix covered the timeout mismatch, the swallowed errors, and the missing dead-letter path together, since fixing only one leaves the loop reachable.
Solution
- Set queue visibility timeout to a safe multiple of the function timeout across every function-queue pairing
- Corrected handler error paths that returned success and caused failed messages to be deleted
- Configured dead-letter queues with a sane maxReceiveCount so poison messages exit the loop
- Enabled partial batch response reporting so one bad record does not retry the whole batch
- Made downstream writes idempotent so any duplicate delivery is harmless
- Added alerting on invocation-to-message ratio and dead-letter queue depth
Outcome
The redrive loop was eliminated, the backlog drained, and duplicate downstream records stopped. Invocation counts returned to tracking message volume, which removed the associated cost spike.
Technologies
AWS LambdaAmazon SQS
Docker
PostgreSQL
Redis
Kafka
RabbitMQ
Kubernetes
Terraform
