Back to all use cases
Healthcare / BenefitsConsultingHybrid

Getting off Pentaho without rewriting everything at once

HP
Healthcare Payer Organization

Overview

Most Pentaho conversations we have now are exit conversations. Teams are carrying hundreds of Kettle transformations that only a shrinking group of people understand, on a runtime that is increasingly hard to staff and license, feeding a warehouse that could do most of the transformation work itself.

Challenge

The organization had roughly four hundred transformations and jobs accumulated over more than a decade. Business logic lived inside step configurations rather than in code, scheduling was spread across Kettle jobs and external cron, and the two engineers who knew the estate best were near retirement. A full rewrite had been estimated in years and repeatedly deferred.

Environment

Pentaho Data Integration on-premises with a Carte cluster, feeding an on-premises warehouse and a growing set of cloud targets.

Approach

AceMQ pushed for a staged migration organized by data domain rather than by transformation. We ran extraction and transformation in parallel, comparing outputs row for row until each domain matched, then decommissioned the Pentaho path for that domain only. Extraction moved to a modern ingest tool; transformation logic moved into warehouse-native SQL models where it belongs.

Solution

  • Parsed the KTR and KJB files to build a dependency graph and rank transformations by downstream impact
  • Grouped the estate into migration waves by data domain, not by file
  • Moved extraction to a modern ingest layer and reimplemented transformation logic as warehouse SQL models
  • Ran old and new pipelines in parallel with automated row-level and aggregate reconciliation per wave
  • Retired Pentaho jobs per domain only after reconciliation held clean through a full reporting cycle
  • Trained the internal team on the new stack during the migration rather than after it

Outcome

The first waves cut over with reconciliation clean and no reporting gaps, and the remaining Pentaho footprint shrank to a well-defined set of domains with a scheduled end date. Critically, the logic ended up in version-controlled SQL that the whole team can read.

Technologies

PentahoAirbytedbtPostgreSQLKubernetes

Ready to Get Started?

Whether you need architecture advisory, 24/7 support, or full managed services, AceMQ has the expertise to help.

Contact Us