Change Data Capture

Change Data Capture: CDC Pipelines Designed, Built and Run

Last reviewed by Tyler Eastridge

Change data capture (CDC) streams every insert, update and delete from a database's transaction log to other systems as it happens, without polling the tables or adding triggers, which is how trigger-based CDC writes changes into extra tables inside the source database. AceMQ designs the pipeline, builds it with Debezium Server into RabbitMQ or with Kafka as the target, tunes the consumers so events stay in order and bad ones do not block the queue, and then supports or runs it on a four-tier ladder.

min fastest response target, by support plan
15
hour to answer a quote request
1
tiers of cover after go-live
4

Is This You?

You probably need this if…

  1. 01

    Nightly batch extracts are the only way other systems learn what changed, and the business wants minutes, not a day

  2. 02

    Triggers or a last_updated column feed a sync job that misses deletes and slows the source database

  3. 03

    A CDC connector exists, but nobody can say what happens to it when the broker or the database restarts

  4. 04

    Change events reach a consumer out of order, and a row ends up in a state it was never in

  5. 05

    One malformed event blocks a queue, or loops forever, and downstream data goes stale behind it

  6. 06

    A Kafka cluster was built for a CDC pipeline that moves a handful of tables to a few consumers

  7. 07

    The transaction log fills the disk, or is purged during an outage, and the pipeline has to start again with a full snapshot

Outcomes

Where you are now, and where you end up

Concrete state changes, not deliverable counts. This is what actually differs about your Infrastructure estate when the engagement closes.

  1. Before

    Downstream systems learn about changes from a nightly job.

    After

    A log-based CDC pipeline that publishes each committed change as an event, with a stated delivery guarantee and a named owner for each stage.

  2. Before

    Events arrive out of order and nobody knows why.

    After

    Consumer and queue settings that preserve order where it matters, with prefetch, redelivery and single-consumer behaviour tested rather than assumed.

  3. Before

    One poison event stalls the pipeline.

    After

    Dead-lettering with a delivery limit, an alert on the dead-letter queue and a documented way to inspect, fix and replay what lands there.

  4. Before

    Recovery means a call to the one person who built it.

    After

    Runbooks for restart, offset and snapshot handling that your own engineers can follow, then cover from the engineers who wrote them.

Recognise your estate here? Get a quote for Infrastructure change data capture from a senior engineer.

Contact Us

Scope

What's covered

  • 01

    CDC Review and Design

    Sources, tables, volumes, consumers and delivery needs read against the options below, ending in a written recommendation and a plan your DBAs can review.

  • 02

    Source Database Preparation

    Log settings, retention, replication slots or per-table capture, and the change list your DBAs need to approve, proven in a test environment before production.

  • 03

    Debezium Server and Kafka Connect

    Debezium Server into RabbitMQ where Kafka is more than the job needs, Kafka Connect where Kafka already runs, with offsets and schema history on storage that survives a restart.

  • 04

    Broker and Consumer Design

    Exchanges or streams per table, queue types, prefetch, dead-lettering, delivery limits and idempotent consumers, designed together because most CDC faults sit between them.

  • 05

    Monitoring and Tracing

    Lag, slot and log retention alerts, dead-letter queue depth, and a trace identifier that lets you follow one row change from the database to the consumer's log.

  • 06

    Runbooks and Operations

    Snapshots, schema changes, upgrades and recovery written for the people on the pager, then support or managed operation on the tier you choose.

Want a senior Infrastructure engineer to scope this with you? Get a quote or send us a message.

Contact Us

The Engagement

How it actually runs

Every phase has a defined duration and a concrete artifact handed over at the end of it. You always know what stage you're in and what you've received.

  1. Phase 1Scoped per source

    CDC Review

    We read each source database's log settings and retention, the tables and change rates to capture, the consumers and what each needs from an event. The output is a written recommendation on method, transport and ownership, with the reasoning attached.

    You receive

    • Source-by-source findings
    • Recommended method and transport with reasons
    • Change list for your DBAs
  2. Phase 2Within a week of the review

    Design

    Event shape, exchange or topic layout, queue types, ordering rules, retry and dead-letter policy, and where offsets and schema history will live. The failure modes to rehearse are agreed before anything is built.

    You receive

    • Reference design
    • Ordering and retry rules per consumer
    • Failure test plan
  3. Phase 3Scoped per engagement

    Build and Failure Tests

    We configure the connectors and the broker with your team, run the initial snapshot without disrupting production, then break it on purpose: broker restart, connector restart, source failover and a poison event.

    You receive

    • Working pipeline in your environment
    • Test results for each failure
    • Fixes verified
  4. Phase 4Ongoing

    Handover and Cover

    Runbooks for snapshots, offsets, schema changes and recovery, a handover session, and the support tier you choose, from on-call escalation to AceMQ running the pipeline.

    You receive

    • Runbooks
    • Agreed support tier and escalation path
    • Handover session with your engineers

Change data capture methods and CDC tools compared

There are four ways to find out what changed in a database. Log-based CDC reads the transaction log, so it sees every committed insert, update and delete, including deletes, in commit order, with little load on the tables. The other three each give something up. Most CDC tools, open source or managed, are a packaged form of one of them.

CDC approaches, example tools and where each fits
ApproachHow it finds changesExample toolsFits whenWatch for
Log-based, self-runReads the database's transaction log (WAL, binlog, SQL Server change tables) and emits one event per row change.Debezium on Kafka Connect or Debezium ServerServices and brokers need events in commit order, and you want to own the pipeline.Log retention and slot growth, snapshots, offsets and schema history. You operate it.
Log-based, managedThe same log reading, run as a cloud service or SaaS that loads a target.AWS DMS, Google Datastream, Fivetran, Striim, Qlik Replicate, Oracle GoldenGateThe consumer is a warehouse or a replica and you want someone else to run capture.Pricing model, which sources and targets are supported, and how much event shape you control.
Trigger-basedDatabase triggers write each change to a shadow table that a job reads.Hand-built, or older replication productsA source with no readable log and a small number of tables.Every write pays for the trigger. Schema changes break it.
Query-basedA job selects rows changed since a timestamp or version column.ELT sync jobs, scheduled SQLA latency of minutes or hours is fine and deletes are tracked another way.Misses hard deletes and intermediate states. Needs a reliable change column.

Native logical replication in PostgreSQL or MySQL replication is a fifth option when the target is another instance of the same database. It is often the right answer for a single source and a single destination, and we say so when the review shows it.

CDC tools compared, vendor by vendor

The right CDC tool depends on where the changes must go. Debezium suits services and brokers that need row changes in commit order and want to run the pipeline themselves; managed tools such as AWS DMS, Qlik Replicate, Striim and Fivetran HVR suit warehouse and replica targets where someone else should run capture. The table lists only what each vendor states on its own pages, checked on 17 November 2026.

CDC tools compared, vendor by vendor (vendor pages checked 17 November 2026)
ToolWhat it isLicence or modelCapture and sources the vendor namesLimit the vendor states
DebeziumOpen-source distributed platform for change data capture, turning committed row changes into eventsApache License 2.0; a Commonhaus Foundation projectStable connectors for MongoDB, MariaDB, MySQL, PostgreSQL, SQL Server, Oracle, Db2, Cassandra and CockroachDB; runs on Kafka Connect, as standalone Debezium Server, or embeddedIncubating connectors are preview releases that may change in ways that are not always backward compatible
AWS Database Migration ServiceAWS service whose tasks capture ongoing changes, as full load plus CDC or CDC onlyAWS managed serviceReads each engine's logs: Oracle LogMiner or binary reader, SQL Server MS-Replication or MS-CDC, MySQL row-based binlogs, PostgreSQL logical replication slotsAWS states that DMS CDC does not provide real-time replication, has no SLAs for CDC latency, and that latency can reach several minutes or longer
Qlik ReplicateData replication software with log-based, zero-footprint captureCommercial: Qlik Cloud Data Integration (Qlik-managed) or client-managed Qlik Data Integration; no prices on the product pageSources include Oracle, SQL Server, Db2, MySQL, Sybase and PostgreSQL; targets include Kafka, Confluent, Snowflake and DatabricksThe vendor's list is not complete; check its support matrix
Fivetran HVRFivetran's replication software for databases and file systems (HVR 6), installed and run in your own environmentFivetran product; the documentation does not state licence or pricingLog-based capture recommended for its lower impact on database resources; trigger-based capture deprecated since 6.2.0/0The NoTruncate capture parameter is not supported for SQL Server
AirbyteData replication platform whose database CDC reads the logs from the last synced positionAirbyte Core under the Elastic License 2.0 (ELv2) from version 0.30.0; Airbyte's own connectors are open sourceCDC sources: Postgres, MySQL, SQL Server, MongoDB, Oracle, SAP HANA and Db2CDC incremental needs primary keys for most sources; only inserts, updates and deletes are captured, not TRUNCATE or ALTER; syncs must run often enough to consume the logs
StriimUnified data integration and streaming platformCommercial: Striim Cloud, Striim Platform and a Striim Developer sandbox; usage-based pricing, no figures on the product pageCDC named for real-time ETL; the product page does not state the capture methodNone stated on the product page
Oracle GoldenGateSoftware package for replication in heterogeneous environments, including transactional change data captureOracle commercial software; licence terms are in Oracle's GoldenGate licensing documentationDocumentation lists requirements for Db2, MySQL, Oracle, PostgreSQL, SQL Server, Sybase, Teradata and TimesTenNone stated on the documentation home page
Confluent CDC connectorsFully managed Kafka Connect source connectors on Confluent CloudConfluent Cloud managed connectorsPostgreSQL, MySQL and SQL Server CDC Source V2 (Debezium), plus Oracle CDC and Oracle XStream CDC sourcesThe older Debezium-based v1 connectors are superseded by the V2 connectors; the legacy PostgreSQL page states one task per connector

Sources, checked 17 November 2026: debezium.io (home page and connector index); AWS DMS User Guide, "Creating tasks for ongoing replication"; qlik.com Qlik Replicate product page; Fivetran HVR 6 documentation and its Capture action reference; Airbyte documentation on CDC and its licence FAQ; striim.com product page; Oracle GoldenGate 26 documentation home; Confluent Cloud connector overview and the PostgreSQL CDC Source (Debezium) page.

How log-based CDC works in SQL Server, PostgreSQL and MySQL

Each database exposes its transaction log differently, and the source database needs a few settings changed before a connector can read it. Those changes usually need a DBA's approval, so we write the change list and prove it in a test environment first.

  • SQL Server — Change data capture is enabled per table, which creates a capture instance and a change table. A capture job reads the transaction log and fills the change table, and the SQL Server Agent must be running for it to work. Debezium reads those change tables.
  • PostgreSQL — Logical decoding turns the write-ahead log into row changes. It needs wal_level set to logical, a replication slot per consumer and an output plugin such as pgoutput, which is built in. The slot keeps WAL until the consumer confirms it, so an unattended slot can fill the disk.
  • MySQL — The connector reads the binary log, which must be in row format with full row images, and the log must be kept longer than the longest outage the pipeline has to survive.

Log-based capture reads the log instead of querying tables, so its load on the source is usually lower than triggers or polling, but it is not zero. SQL Server writes and cleans change tables inside the database, and a PostgreSQL slot holds disk. We size both before go-live.

Ready to talk Infrastructure? Get a quote or send us a message.

Contact Us

Where the changes go: RabbitMQ or Kafka

The capture side and the transport are separate decisions, and the transport should follow the consumers.

  • RabbitMQ — Fits when services need routed events and each consumer reads from its own queue. Debezium Server publishes to an exchange with a routing key taken from the table or the record key, and an exchange fans one change out to as many queues as needed. There is no cluster of brokers, Connect workers and a registry to run for a handful of tables.
  • Kafka — Fits when Kafka is already your backbone, when many sources feed many consumers, or when consumers must replay history from an offset. Debezium then runs on Kafka Connect. The Kafka cluster and the Connect cluster become part of what you operate.
  • RabbitMQ Streams — A middle path for replay and large fan-out without a Kafka cluster, through the Debezium Server stream sink.
  • Neither — If the consumer is a warehouse and nothing else needs the events, a managed CDC service or an ELT tool is simpler than any broker.

Debezium Server runs exactly one connector per instance, so several sources mean several instances. Its offsets default to a local file, which is lost on a container's ephemeral disk, so where they live is a design decision made before the first snapshot.

Running CDC consumers: what goes wrong and how to prevent it

Most CDC incidents are not in the capture. They are in the queue and consumer behind it. These are the settings we check first on a RabbitMQ-based pipeline.

How we set prefetch on ordered consumers

A change event is only safe to apply if the events before it for the same row have been applied. RabbitMQ round-robins messages across consumers on a queue unless the queue is declared as single active consumer, and a consumer that holds a batch of unacknowledged messages and then fails hands that whole batch back for redelivery. In our field experience a prefetch of 1 on the queue of an ordered CDC consumer kept events in order where a larger prefetch did not. Pair it with one active consumer per queue, or with a routing key that sends one table's events to one consumer. This is our practice rather than a RabbitMQ rule: a prefetch of 1 trades throughput for a smaller in-flight window, so test it against your own load.

Dead-letter poison events with a delivery limit

A poison event is one that a consumer can never process, such as a payload it cannot parse. Without a limit it is redelivered forever and everything behind it waits. On quorum queues the delivery limit defaults to 20 from RabbitMQ 4.0, and a message that reaches it is dropped, or dead-lettered if a dead-letter exchange is configured. Set the exchange, and use the at-least-once dead-letter strategy, which requires the reject-publish overflow setting and can produce duplicates at the target. Alert on the dead-letter queue and write down how someone inspects, fixes and replays what lands there.

A change in RabbitMQ 4.3

From RabbitMQ 4.3 the delivery limit counts delivery-count, so a consumer that negatively acknowledges an event with requeue no longer moves it towards the limit, while a reject does. A consumer that nacks a poison event in a loop can spin indefinitely. Reject it so it dead-letters, or route it explicitly.

Plan for the connector to stop and restart

When the broker or the database goes away, a connector can stop. The pipeline survives if the process is restarted by its supervisor, such as Kubernetes, and if its offsets sit on storage that persists, because the connector resumes from the last committed offset. Offsets are committed periodically, so after an outage some events can arrive twice. Delivery is at least once, not exactly once, and consumers must be idempotent on the primary key. On PostgreSQL the same holds inside the database: a slot's position is persisted at checkpoint, so after a crash recent changes can be sent again.

Hold the source log long enough

A connector that is down needs the database to keep its log. If the log is purged during an outage there is no position to resume from and the answer is a new snapshot. Set retention longer than the longest outage you must survive, and alert before it runs out. On PostgreSQL an unconsumed replication slot does the opposite and keeps WAL until the disk fills, so slot lag needs an alert as well.

Trace one change from end to end

Stamp each event with an identifier that travels in a message header and appears in the consumer's logs. When a row is wrong downstream, a trace identifier turns a search across three systems into one query. Time-ordered identifiers such as UUID version 7 also sort by creation, which helps when reading logs.

Recognise your estate here? Get a quote for Infrastructure change data capture from a senior engineer.

Contact Us

Debezium vs managed CDC

Choose Debezium when the consumers are services, queues or streams that need ordered row changes and you want control of the event shape; choose a managed CDC service when the only consumer is a warehouse or replica and you would rather pay a vendor to run capture. Both read the database log, so the difference is who runs it, where the events land and how much you control.

Debezium against managed CDC services
QuestionDebeziumManaged CDC (DMS, Qlik Replicate, Striim, Fivetran HVR)
Who runs captureYou or AceMQ, on Kafka Connect or Debezium ServerThe vendor's service or product, with its connectors
Where events goKafka, or through Debezium Server to RabbitMQ, RabbitMQ Streams and other destinationsMostly warehouses, replicas and the streaming targets the vendor supports
Event shapeOne event per row change, shaped by connector configuration and single message transformsThe vendor's format; check how much routing you can control
Cost modelNo licence fee; you pay for the hosts, brokers and the people who operate themSubscription or usage; Striim and Qlik show no prices on their product pages
What you operateLog retention and slot growth, snapshots, offsets, schema history, connector restartsLess, but you still need log retention on the source and monitoring of lag
LatencyMeasured in your environmentAWS states DMS CDC is not real-time replication and has no SLAs for CDC latency
ExitOpen source and open event formatsDepends on the vendor; check source and target lists before committing

Where AceMQ fits

AceMQ designs the Debezium pipeline, runs it into RabbitMQ or Kafka, tunes the consumers so events stay in order and a bad event cannot block the queue, and supports or operates it on the four-tier ladder. We configure and fix connectors; we do not write new ones. For a warehouse-only load the review will say a managed service is cheaper. The worked example below shows a standalone Debezium deployment publishing to RabbitMQ in place of Kafka.

A worked example: Debezium Server into RabbitMQ at American National

American National Insurance needed CDC for SQL Server on Kubernetes and evaluated Kafka. Running Kafka and Kafka Connect was out of proportion to the volume and routing the pipeline needed, so Kafka was judged too complex for the job. AceMQ assessed the requirement and designed a standalone Debezium deployment that publishes change events straight to RabbitMQ exchanges, keeping SQL Server change capture and adding more flexible routing.

Want a senior Infrastructure engineer to scope this with you? Get a quote or send us a message.

Contact Us

After go-live: four tiers of cover

Operational tiers for a CDC pipeline
TierWhat it isThe question it answers
Tier 1: Reactive supportSenior engineers on call for incidents under a contracted SLA, with response targets from 15 minutes, by plan.Change events have stopped. Who joins the call?
Tier 2: Proactive supportTier 1 plus monitoring of lag, log retention, slot growth and dead-letter queues.Who notices the lag before the business does?
Tier 3: Managed on your infrastructureAceMQ operates the pipeline inside your cloud account or data center.The data must stay inside our boundary and we do not want the pager.
Tier 4: Managed on AceMQ's infrastructureAceMQ hosts and operates the broker and the capture.We want it run for us, and our policy allows a hosted service.

Who this is not for, and what we do not promise

  • Warehouse-only loads — If the only consumer is a warehouse and nobody else needs the events, a managed CDC service or an ELT tool will be cheaper than a broker pipeline. We will say so in the review.
  • One database, one copy — A single PostgreSQL or MySQL source feeding one replica is usually better served by the database's own replication.
  • Exactly-once delivery — We do not promise it. CDC delivery is at least once, and the design makes consumers idempotent.
  • Latency figures in advance — End-to-end delay depends on the source log, the connector, the broker and the consumers. We measure it in your environment and do not quote a number beforehand.
  • Connector development — We configure, run and fix Debezium and Kafka Connect connectors. We do not write new connectors.
  • Oracle, Db2 and MongoDB sources — Oracle, Db2 and MongoDB sources are scoped with your DBAs, with specialist partners where needed, and proven in a test environment before production.

Ready to talk Infrastructure? Get a quote or send us a message.

Contact Us

Customer Success

Real Infrastructure Results

See how enterprises trust AceMQ for their most critical Infrastructure workloads.

1 / 8 case studies

  • Consulting

    Choosing Debezium + RabbitMQ over Kafka for Change Data Capture

    American National Insurance

    American National Insurance chose a standalone Debezium + RabbitMQ architecture for SQL Server change data capture after judging Kafka too complex to run for the job, keeping operations lean and routing flexible.

    KafkaRabbitMQDebeziumKubernetes
  • Support

    PostgreSQL WAL and Replication Slot Support

    Digital Banking Platform

    Ongoing support for a cluster where an abandoned logical replication slot repeatedly filled the WAL volume and threatened to halt the primary.

    PostgreSQLDebeziumKafkaPrometheus
  • Remediation

    Debezium Oracle LogMiner Lag Remediation

    Regional Retail Bank

    Bringing an Oracle change data capture pipeline back inside its batch window after nightly jobs left the Debezium connector hours behind the source.

    DebeziumOracleKafkaKubernetes
  • Support

    Debezium MySQL Binlog Retention and Schema History Support

    Online Travel Platform

    Ongoing support for a MySQL change data capture pipeline that needed a full re-snapshot after a planned outage outlasted binlog retention and a trimmed schema history topic.

    DebeziumMySQLSQL ServerOracle DatabaseKafkaAmazon RDS
  • Assessment

    MongoDB to PostgreSQL JSONB Assessment

    SaaS Platform Team

    A typical engagement for a team that wants to consolidate on PostgreSQL: documents, aggregation pipelines, transactions and change streams assessed against a JSONB design, with a stay, proxy or leave recommendation.

    MongoDBPostgreSQLDebezium
All use cases

FAQ

Questions about Infrastructure change data capture

What is change data capture?

Change data capture is a way of finding out what changed in a database and sending each change to other systems as it happens. Log-based CDC does this by reading the database's transaction log, so it captures inserts, updates and deletes in commit order without polling the tables or adding triggers.

What is the difference between CDC and ETL?

ETL and ELT move data in scheduled batches, usually into a warehouse. CDC moves each change as it is committed, so services and queues can react in near real time. Many teams use both: CDC to capture changes, and a loader to land them in a warehouse.

Which CDC tools should we use?

It depends on the consumer. Services and brokers fit log-based tools such as Debezium. Warehouses fit managed services such as AWS DMS, Google Datastream or Fivetran. A single PostgreSQL or MySQL copy can use native replication. We compare them for your sources in the review.

What is the best CDC tool?

There is no best tool, only a best fit for the consumer. Log-based Debezium fits services and brokers that need row changes in commit order. AWS DMS, Qlik Replicate, Striim and Fivetran HVR fit warehouse and replica targets when you want the vendor to run capture. A single PostgreSQL or MySQL copy can use the database's native replication. The review compares them against your sources, targets and event-routing needs.

Is Debezium better than AWS DMS?

For event-driven services, usually yes: Debezium gives you the events in commit order and the routing control, and it is open source under the Apache License 2.0. For a one-off or warehouse load, DMS is simpler to start. AWS states that DMS CDC does not provide real-time replication and that there are no SLAs for CDC latency, so check that against your requirement.

Does CDC need Kafka?

No. Debezium usually runs on Kafka Connect, but Debezium Server runs the same connectors as a standalone application and can publish to RabbitMQ, RabbitMQ Streams and other destinations. Kafka is the better fit when many consumers need to replay history from an offset.

Does change data capture slow down the database?

Log-based CDC reads the transaction log rather than querying tables, so its load is usually lower than triggers or polling, but not zero. SQL Server maintains change tables inside the database, and a PostgreSQL replication slot holds WAL on disk. Delivery is at least once, so consumers must also handle duplicates.

Let's talk

Talk to an Engineer About Your CDC Pipeline

Tell us which databases you capture from, where the changes go and what has gone wrong. We will scope a CDC review and reply to your quote request within 1 hour.

Email Us

Get in Touch

Talk to an AceMQ Expert

Send us a message and we'll follow up within 1 hour — or book a free 30-min consultation directly.

305-204-2607
info@acemq.com
66 W. Flagler St. 9th Floor
Miami, FL 33130

Prefer to talk now? Call us directly or use the consultation tab to find a time that works.

We respond within 1 hour.

Pick a time that works — no pressure, no pitch. Just 30 minutes with an expert.

We respond within 1 hour.