Choosing the right event source before you write the handler

Overview

The hard decisions in a Lambda architecture are made at the event source, not in the handler: queue versus stream, batch size and window, ordering guarantees, concurrency limits, and what happens to a message that cannot be processed. Getting these wrong produces problems that look like application bugs.

Challenge

The estate mixed queues and streams with no clear rule for which to use. Ordering-sensitive workloads ran on sources that gave no ordering guarantee, batch sizes were left at defaults regardless of per-record cost, unbounded concurrency on some functions was overwhelming a downstream legacy system, and failure handling varied per function so operators had no consistent way to recover.

Environment

AWS Lambda consuming queues, streams, and messaging systems, integrating a mix of modern services and a capacity-limited legacy backend. AceMQ designs the customer's event architecture on the platform; AWS operates the platform.

Approach

AceMQ classified each workload by ordering requirement, throughput, per-record processing cost, and downstream capacity, then mapped each class to an appropriate event source with defined batching, concurrency, and failure semantics. Concurrency limits were set explicitly where a downstream system has a hard ceiling, because that is a design decision rather than a tuning afterthought.

Solution

• Classified workloads by ordering requirement, throughput, per-record cost, and downstream capacity
• Defined which event source each workload class should use, with the rationale documented
• Set batch size and batching window per workload rather than leaving platform defaults in place
• Applied reserved concurrency to protect the capacity-limited legacy backend from bursts
• Standardized failure handling with dead-letter destinations and on-failure destinations across the estate
• Established an idempotency pattern so at-least-once delivery is safe by default

Outcome

Technologies