Overview
Airbyte's most damaging failure mode is not a red job — it is a green one. When a source adds, renames, or retypes a column and the connection's schema propagation policy silently ignores it, records keep landing but the columns the business actually reports on stop updating.
Challenge
A source application shipped a release that renamed two columns and changed a numeric field to a string. The Airbyte connection continued to complete without error, but downstream models built on those fields returned nulls. Nobody noticed for several reporting cycles because every dashboard alert was wired to job status rather than data freshness.
Environment
Self-hosted Airbyte on Kubernetes syncing PostgreSQL and SaaS sources into a cloud warehouse, with dbt models consuming the raw layer.
Approach
AceMQ engineers pulled the connection's catalog history and compared the discovered schema against the configured stream schema to pinpoint exactly which fields had drifted and when. From there we traced the destination normalization behavior, corrected the schema change policy, and backfilled the affected windows in controlled batches rather than a single full refresh.
Solution
- Diffed discovered source catalogs against configured stream schemas to identify drifted and dropped fields
- Reset the schema change propagation policy per stream so breaking changes fail loudly instead of silently truncating
- Rebuilt affected streams with corrected primary keys and cursor fields
- Backfilled the impacted date ranges in bounded batches to avoid warehouse credit spikes
- Added freshness and row-count checks at the destination so data staleness alerts independently of job status
Outcome
The stale columns were repopulated and the connection now fails visibly when a source schema changes in a way it cannot absorb. The team stopped relying on job status as a proxy for data correctness.
Technologies
Airbyte
PostgreSQL
Snowflake
Kubernetes
dbt
Kafka
