An honest inventory of what your Airbyte estate is actually doing

Overview

Airbyte estates tend to accumulate. Connections get added quickly because they are easy to add, and after a year nobody can say which connectors are pinned to old versions, which streams are on full refresh for no reason, or which pipelines nothing downstream consumes.

Challenge

The data platform team had over a hundred connections across several workspaces. Connector versions ranged widely, custom connectors had been built by people who had since left, and there was no consistent pattern for primary keys, deduplication, or dead-letter handling. Nobody could estimate the blast radius of a connector upgrade.

Environment

Self-hosted Airbyte on Kubernetes with a mix of certified, community, and internally built connectors feeding a cloud warehouse.

Approach

AceMQ inventoried every connection through the Airbyte API and mapped each stream to its downstream consumers, then scored connections on version currency, sync mode fitness, state reliability, and observability. The output was a ranked remediation backlog rather than a generic maturity score.

Solution

• Full inventory of workspaces, connections, streams, and connector versions via the Airbyte API
• Mapped each stream to downstream warehouse models to identify unused and orphaned pipelines
• Flagged streams where the chosen sync mode does not match the table's change characteristics
• Reviewed custom connector code for retry, pagination, and rate-limit handling gaps
• Delivered a prioritized backlog with effort estimates and an upgrade sequencing plan

Outcome

Technologies