Getting off Pentaho without rewriting everything at once
Airbytedbt
PostgreSQL
KubernetesThe first waves cut over with reconciliation clean and no reporting gaps, and the remaining Pentaho footprint shrank to a well-defined set of domains with a scheduled end date. Critically, the logic e…
Overview
Most Pentaho conversations we have now are exit conversations. Teams are carrying hundreds of Kettle transformations that only a shrinking group of people understand, on a runtime that is increasingly hard to staff and license, feeding a warehouse that could do most of the transformation work itself.
Challenge
The organization had roughly four hundred transformations and jobs accumulated over more than a decade. Business logic lived inside step configurations rather than in code, scheduling was spread across Kettle jobs and external cron, and the two engineers who knew the estate best were near retirement. A full rewrite had been estimated in years and repeatedly deferred.
Environment
Pentaho Data Integration on-premises with a Carte cluster, feeding an on-premises warehouse and a growing set of cloud targets.
Approach
AceMQ pushed for a staged migration organized by data domain rather than by transformation. We ran extraction and transformation in parallel, comparing outputs row for row until each domain matched, then decommissioned the Pentaho path for that domain only. Extraction moved to a modern ingest tool; transformation logic moved into warehouse-native SQL models where it belongs.
Solution
- 1Parsed the KTR and KJB files to build a dependency graph and rank transformations by downstream impact
- 2Grouped the estate into migration waves by data domain, not by file
- 3Moved extraction to a modern ingest layer and reimplemented transformation logic as warehouse SQL models
- 4Ran old and new pipelines in parallel with automated row-level and aggregate reconciliation per wave
- 5Retired Pentaho jobs per domain only after reconciliation held clean through a full reporting cycle
- 6Trained the internal team on the new stack during the migration rather than after it
Outcome
The first waves cut over with reconciliation clean and no reporting gaps, and the remaining Pentaho footprint shrank to a well-defined set of domains with a scheduled end date. Critically, the logic ended up in version-controlled SQL that the whole team can read.
Technologies
AirbytedbtPostgreSQLKubernetesRelated Use Cases
Pentaho Lineage and Documentation Recovery Assessment
Reconstructing lineage and documentation for an undocumented estate of legacy Kettle transformations before anyone attempts to change or replace them.
Airbyte Pipeline and Connector Estate Assessment
A structured review of an Airbyte estate that had grown organically — auditing connector versions, sync modes, state handling, and failure visibility.
Need Pentaho Architecture Guidance?
AceMQ's senior Pentaho engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.