Choosing the right event source before you write the handler
AWS Lambda
Amazon SQSKafka
RabbitMQ
DockerThe legacy backend stopped being overwhelmed by burst traffic, ordering-sensitive workloads moved onto sources that actually guarantee ordering, and failure recovery became one documented procedure in…
Overview
The hard decisions in a Lambda architecture are made at the event source, not in the handler: queue versus stream, batch size and window, ordering guarantees, concurrency limits, and what happens to a message that cannot be processed. Getting these wrong produces problems that look like application bugs.
Challenge
The estate mixed queues and streams with no clear rule for which to use. Ordering-sensitive workloads ran on sources that gave no ordering guarantee, batch sizes were left at defaults regardless of per-record cost, unbounded concurrency on some functions was overwhelming a downstream legacy system, and failure handling varied per function so operators had no consistent way to recover.
Environment
AWS Lambda consuming queues, streams, and messaging systems, integrating a mix of modern services and a capacity-limited legacy backend. AceMQ designs the customer's event architecture on the platform; AWS operates the platform.
Approach
AceMQ classified each workload by ordering requirement, throughput, per-record processing cost, and downstream capacity, then mapped each class to an appropriate event source with defined batching, concurrency, and failure semantics. Concurrency limits were set explicitly where a downstream system has a hard ceiling, because that is a design decision rather than a tuning afterthought.
Solution
- 1Classified workloads by ordering requirement, throughput, per-record cost, and downstream capacity
- 2Defined which event source each workload class should use, with the rationale documented
- 3Set batch size and batching window per workload rather than leaving platform defaults in place
- 4Applied reserved concurrency to protect the capacity-limited legacy backend from bursts
- 5Standardized failure handling with dead-letter destinations and on-failure destinations across the estate
- 6Established an idempotency pattern so at-least-once delivery is safe by default
Outcome
The legacy backend stopped being overwhelmed by burst traffic, ordering-sensitive workloads moved onto sources that actually guarantee ordering, and failure recovery became one documented procedure instead of a per-function investigation.
Technologies
AWS LambdaAmazon SQSKafkaRabbitMQDockerRelated Use Cases
AWS Lambda Cost and Right-Sizing Assessment
Measuring memory, duration, and concurrency across a large Lambda estate to right-size functions that were provisioned by guesswork.
AWS Lambda SQS Redrive Loop Remediation
Breaking a redrive loop where SQS messages were reprocessed indefinitely because the queue visibility timeout was shorter than the Lambda function timeout.
Need AWS Lambda Architecture Guidance?
AceMQ's senior AWS Lambda engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.