Hung Carte slaves, leaked memory, and transformations that never come back

Overview

Clustered Pentaho execution moves work onto Carte slave servers, and when those slaves degrade the symptom is a transformation that simply never finishes. The master reports it as running indefinitely, and the operator's only lever is a restart.

Challenge

Clustered transformations began hanging several times a week. Carte slave JVMs grew steadily through the day and were never reclaimed, orphaned transformation sessions accumulated in the slave's registry, and restarting the master did not clear them. The batch window began overrunning into the production shift.

Environment

Pentaho Data Integration with a Carte master and several slave servers on on-premises Linux hosts, feeding a manufacturing reporting warehouse.

Approach

AceMQ captured heap dumps and thread dumps from a hung slave rather than restarting it, which identified where rows were backing up and which step was blocking. We corrected the row-set sizing that was causing unbounded buffering, added lifecycle handling so completed transformations are cleaned from the slave registry, and containerized the slaves so a degraded instance is replaced rather than nursed.

Solution

• Captured heap and thread dumps from hung Carte slaves to identify the blocking step and buffering behavior
• Corrected row-set sizes and partitioning so clustered steps stop buffering without bound
• Added cleanup of completed and orphaned transformation sessions from the slave registry
• Containerized Carte slaves with health checks so degraded instances are replaced automatically
• Instrumented slave JVM memory and active-transformation counts with alerting thresholds

Outcome

Technologies