Overview
A marketplace operator sent order events through an Amazon SQS FIFO queue to a fleet of containerised workers that updated inventory and notified sellers. At peak the queue backed up while the workers sat mostly idle, and twice a month it stopped moving altogether until an engineer purged part of it by hand.
Challenge
Every message was sent with the same MessageGroupId, chosen to guarantee global ordering. SQS orders FIFO messages within a group and holds back later messages in that group while an earlier one is in flight, so the whole queue behaved as one stream and extra workers added nothing. The stalls had a second cause. A malformed message failed every time it was received, came back after the visibility timeout, and sat at the head of the one group. The redrive policy allowed so many receives that it would have taken days to reach the dead-letter queue.
Environment
An Amazon SQS FIFO queue with a dead-letter queue, consumed by Java workers on Amazon ECS with AWS Fargate. Producers were order services sending one message per API call.
Approach
AceMQ started from the queue metrics: age of the oldest message, messages in flight, and receive counts on the stuck message. It then asked the owners what ordering the business actually needed, which turned out to be per order and not across all orders, and changed the design to match.
Solution
- Replaced the constant MessageGroupId with the order ID, so ordering holds per order and groups run in parallel
- Turned on high-throughput FIFO with the limit applied per message group
- Switched producers to batched sends of up to ten messages instead of one call per message
- Set maxReceiveCount on the FIFO dead-letter queue to a small number, so a message that cannot be processed leaves its group after a few attempts
- Made the workers idempotent on the order ID and event type, so a redelivery after a visibility timeout is harmless
- Added alarms on age of the oldest message and on dead-letter queue depth, and documented how to inspect and redrive a message
Outcome
The backlog at peak cleared with the existing worker fleet, and a malformed message now leaves its group after a few attempts and lands in the dead-letter queue with an alarm. Nobody has needed to purge the queue by hand since the change.
Technologies
Amazon SQSAWS
Docker
Java
PostgreSQL
Terraform
Kafka
RabbitMQ
Linux
