The same message, processed forever, billed every time

Overview

When an SQS queue's visibility timeout is shorter than the consuming Lambda's timeout, a message becomes visible again while the first invocation is still working on it. A second invocation picks it up, and under sustained load this becomes a self-feeding loop of duplicate work that costs real money and produces duplicate side effects.

Challenge

A queue that normally ran near empty accumulated a large backlog overnight, invocation counts were far higher than message counts, and downstream records showed duplicates. The function timeout had been raised to handle larger payloads without a corresponding change to the queue's visibility timeout. Compounding it, a handler error path returned success, so failed messages were deleted rather than sent to the dead-letter queue, and a separate class of poison messages recirculated indefinitely.

Environment

AWS Lambda functions consuming SQS queues, writing to a relational database and calling internal APIs. AceMQ works on the customer's functions, event source configuration, and integration design — not on the AWS platform itself.

Approach

AceMQ compared invocation counts against message counts to confirm duplication, then audited every function-to-queue pairing for timeout relationships, batch behavior, and error handling. The fix covered the timeout mismatch, the swallowed errors, and the missing dead-letter path together, since fixing only one leaves the loop reachable.

Solution

• Set queue visibility timeout to a safe multiple of the function timeout across every function-queue pairing
• Corrected handler error paths that returned success and caused failed messages to be deleted
• Configured dead-letter queues with a sane maxReceiveCount so poison messages exit the loop
• Enabled partial batch response reporting so one bad record does not retry the whole batch
• Made downstream writes idempotent so any duplicate delivery is harmless
• Added alerting on invocation-to-message ratio and dead-letter queue depth

Outcome

Technologies