When a row ends up in a state it was never in, look at the consumer before the connector

Overview

A change data capture pipeline usually works in testing and fails in production at the queue and the consumer, not at the capture. AceMQ reviews the whole path from the connector's output to the consumer's write and fixes ordering, retries and poison events as one design.

Environment

Debezium Server publishing to RabbitMQ exchanges from a relational database, running on Kubernetes, with quorum queues and several consumer services writing to their own stores.

Approach

AceMQ reads the connector settings, the exchange and queue layout, the consumer code paths and the broker's delivery behaviour for the RabbitMQ version in use. Ordering is addressed first: queue and consumer settings that keep events for one row in sequence. Poison events are then routed to a dead-letter queue with a delivery limit, and restart behaviour is tested with the connector, the broker and the consumer in turn.

Solution

1. 01Consumer prefetch and concurrency review, with a prefetch of 1 and one active consumer per queue where order matters
2. 02Routing keys and queue layout so one table's events reach one consumer
3. 03Dead-letter exchange, delivery limit and at-least-once dead-letter strategy on quorum queues, with the RabbitMQ 4.3 change to how negative acknowledgements count toward the limit taken into account
4. 04A dead-letter queue alert and a written procedure to inspect, correct and replay events
5. 05Idempotent consumer writes keyed on the primary key, because delivery is at least once
6. 06Offsets on persistent storage and a rehearsed restart of the connector, the broker and the consumer
7. 07A trace identifier carried in a message header and logged by each consumer

Technologies

DebeziumRabbitMQKubernetes