Back to all use cases

SupportSoftware / Digital Platforms

A six-hour maintenance window should not cost a fourteen-hour re-snapshot

Online Travel Platform

Ongoing support for a MySQL change data capture pipeline that needed a full re-snapshot after a planned outage outlasted binlog retention and a trimmed schema history topic.

Overview

A travel platform fed bookings from Amazon RDS for MySQL into Kafka with the Debezium MySQL connector. During a planned Kafka Connect upgrade the connector was stopped for several hours. When it started again, the binlog position it needed no longer existed, and the schema history topic had also lost records. The only way back was a full re-snapshot of a large bookings database.

Challenge

RDS for MySQL purges binary logs quickly unless retention is set explicitly, and the Debezium connector needs the binlog from its last committed position. The schema history topic had been created with default retention, so it was trimmed on a schedule. The team found out about both at the same moment, with downstream search and pricing services stale until the snapshot finished.

Environment

Amazon RDS for MySQL 8.0 with row-based binlog; Kafka Connect on Kubernetes on AWS; Debezium MySQL connector; a Kafka cluster with default topic settings; downstream consumers building search and pricing views.

Approach

AceMQ took over as the escalation path for the pipeline. The first job was to make the next outage survivable: retention at the database and on the history topic, and an alert on connector state. The second was to replace the full re-snapshot with a path that backfills only what is missing, and to document when each recovery mode is safe.

Solution

  1. Set RDS binlog retention to cover the longest planned and unplanned outage and added an alert on the oldest binlog age against connector lag
  2. Recreated the schema history topic with one partition, no deletion and a replication factor matching the other Connect topics
  3. Added heartbeats and alerts on connector status and on time since the last event
  4. Introduced incremental snapshots with a signalling table so a table can be backfilled without stopping streaming
  5. Documented the snapshot and recovery modes, including why schema-only recovery is unsafe if the schema changed while the connector was stopped
  6. Rehearsed a Connect upgrade in staging, including a connector stopped for longer than usual, before the next production change

Outcome

The next Kafka Connect upgrade ran with the connector stopped for hours and resumed from its saved position with no snapshot. A backfill of one table now takes a signal rather than an outage, and the team no longer needs to escalate routine connector restarts.

Technologies

DebeziumMySQLKafkaAmazon RDS
  • Support

    PostgreSQL WAL and Replication Slot Support

    Digital Banking PlatformFinancial Services

    Ongoing support for a cluster where an abandoned logical replication slot repeatedly filled the WAL volume and threatened to halt the primary.

    PostgreSQLDebeziumKafkaPrometheus
    Read case study: PostgreSQL WAL and Replication Slot Support
  • Remediation

    MySQL Replication Lag Remediation

    E-Commerce Marketplace OperatorCross-Industry

    Resolving replica lag that grew to hours during batch windows because single-threaded apply could not keep pace with large multi-row transactions.

    MySQLProxySQLPrometheusLinux
    Read case study: MySQL Replication Lag Remediation
  • Consulting

    Replacing Kafka with Debezium + RabbitMQ for Change Data Capture

    American National InsuranceHealthcare

    American National Insurance replaced a Kafka-based CDC pipeline with a standalone Debezium + RabbitMQ architecture, simplifying operations while maintaining SQL Server change capture with improved message routing flexibility.

    KafkaRabbitMQDebeziumKubernetes
    Read case study: Replacing Kafka with Debezium + RabbitMQ for Change Data Capture
  • Remediation

    ClickHouse Too Many Parts Remediation

    Adtech Measurement PlatformSoftware

    Resolving ingestion failures where frequent small inserts produced parts faster than background merges could retire them, tripping the parts limit.

    ClickHouseKafkaKubernetesPrometheus
    Read case study: ClickHouse Too Many Parts Remediation
  • Remediation

    Debezium Oracle LogMiner Lag Remediation

    Regional Retail BankFinancial Services

    Bringing an Oracle change data capture pipeline back inside its batch window after nightly jobs left the Debezium connector hours behind the source.

    DebeziumOracleKafkaKubernetes
    Read case study: Debezium Oracle LogMiner Lag Remediation

Related Reading

Need Expert Debezium Support?

AceMQ's senior Debezium engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.