Finding the sync that reports green while the warehouse table goes stale

Overview

Airbyte's most damaging failure mode is not a red job — it is a green one. When a source adds, renames, or retypes a column and the connection's schema propagation policy silently ignores it, records keep landing but the columns the business actually reports on stop updating.

Challenge

A source application shipped a release that renamed two columns and changed a numeric field to a string. The Airbyte connection continued to complete without error, but downstream models built on those fields returned nulls. Nobody noticed for several reporting cycles because every dashboard alert was wired to job status rather than data freshness.

Environment

Self-hosted Airbyte on Kubernetes syncing PostgreSQL and SaaS sources into a cloud warehouse, with dbt models consuming the raw layer.

Approach

AceMQ engineers pulled the connection's catalog history and compared the discovered schema against the configured stream schema to pinpoint exactly which fields had drifted and when. From there we traced the destination normalization behavior, corrected the schema change policy, and backfilled the affected windows in controlled batches rather than a single full refresh.

Solution

• Diffed discovered source catalogs against configured stream schemas to identify drifted and dropped fields
• Reset the schema change propagation policy per stream so breaking changes fail loudly instead of silently truncating
• Rebuilt affected streams with corrected primary keys and cursor fields
• Backfilled the impacted date ranges in bounded batches to avoid warehouse credit spikes
• Added freshness and row-count checks at the destination so data staleness alerts independently of job status

Outcome

Technologies