Overview

American National Insurance was running a Kafka-based change data capture (CDC) pipeline for SQL Server and evaluating whether Kafka remained the right tool for their messaging architecture. After a thorough assessment, AceMQ designed a replacement architecture using standalone Debezium publishing directly to RabbitMQ.

Challenge

Kafka introduced significant operational complexity for a CDC use case that did not require Kafka's core capabilities. Managing Kafka clusters, Kafka Connect, and associated operational overhead was disproportionate to the actual data volume and routing requirements.

Environment

SQL Server databases; Kafka Connect with Debezium for CDC; Kubernetes deployment; insurance company with compliance requirements.

Approach

AceMQ analyzed the CDC requirements and designed a standalone Debezium connector deployment that publishes change events directly to RabbitMQ exchanges. This eliminated the Kafka cluster dependency while preserving change capture capability and adding more flexible message routing through RabbitMQ's exchange and binding model.

Solution

• Kafka architecture assessment identifying unnecessary complexity for the CDC use case
• Standalone Debezium deployment design publishing directly to RabbitMQ
• SQL Server CDC configuration and connector tuning
• RabbitMQ exchange and binding design for change event routing
• Migration plan and cutover strategy from Kafka to the new architecture

Outcome

Technologies