When incremental quietly becomes full refresh, the warehouse bill tells you first

Overview

Incremental syncs depend on a stable cursor and a preserved connection state. When either is invalidated — by a config change, a connector version bump, or a cursor column that is not monotonic — Airbyte falls back to reading everything, and the cost lands in the warehouse rather than in Airbyte itself.

Challenge

A nightly connection against a multi-hundred-million-row table began running for hours instead of minutes. The cursor column was an updated-at timestamp that the application occasionally backdated, so records were skipped on some runs and the team's fix — resetting the connection — triggered a full re-read each time. Warehouse compute for the ingest window grew several times over.

Environment

Airbyte Cloud and self-managed workers on AWS, PostgreSQL sources, cloud data warehouse destination with usage-based compute pricing.

Approach

AceMQ reviewed the connection state payloads across recent runs to confirm where state was being discarded, then evaluated whether the table was better served by cursor-based incremental or log-based CDC. We moved the highest-volume streams to CDC and left low-churn reference tables on scheduled full refresh, where that is genuinely the cheaper option.

Solution

• Traced per-stream state payloads across runs to identify where incremental state was being invalidated
• Replaced an unreliable timestamp cursor with log-based CDC for the highest-volume tables
• Separated high-churn and low-churn streams into different connections with different schedules
• Tuned sync concurrency and destination batch sizing to smooth warehouse cluster demand
• Documented which config changes trigger a state reset so routine edits stop causing full refreshes

Outcome

Technologies