Overview
American National Insurance needed change data capture (CDC) from SQL Server and weighed Kafka for the job. Kafka was judged too complex to run for this use case, so AceMQ designed an architecture using standalone Debezium Server publishing directly to RabbitMQ, which the team already ran.
Challenge
A Kafka-based CDC pipeline would have meant running Kafka clusters and Kafka Connect for a use case that did not need Kafka's core capabilities. That operational load was out of proportion to the data volume and routing requirements.
Environment
SQL Server databases; existing RabbitMQ estate; Kubernetes deployment; insurance company with compliance requirements.
Approach
AceMQ analyzed the CDC requirements and designed a standalone Debezium connector deployment that publishes change events directly to RabbitMQ exchanges. This avoided a Kafka cluster dependency while delivering change capture and adding more flexible message routing through RabbitMQ's exchange and binding model.
Solution
- Assessment of Kafka versus a standalone Debezium design for the CDC use case
- Standalone Debezium deployment design publishing directly to RabbitMQ
- SQL Server CDC configuration and connector tuning
- RabbitMQ exchange and binding design for change event routing
- Rollout plan for the Debezium Server deployment on Kubernetes
Outcome
American National Insurance got SQL Server change capture without taking on Kafka's operational overhead, with flexible message routing through the RabbitMQ estate it already ran.