Back to all use cases

SupportCross-Industry

24/7 senior engineers, a status-first recovery decision and a written root cause after every P1

Data Platform Team

A typical 24/7 support engagement for failed or hung PowerCenter workflows: a live engineer on the incident, a status-first decision to recover, restart or rerun, a check of what the targets hold, and a written root-cause analysis.

Informatica support

Overview

This is a typical engagement for a data team whose nightly PowerCenter load feeds reporting that finishes before the business day starts. The team keeps its Informatica support contract for product fixes and adds AceMQ for the hours and the operating questions it does not cover. AceMQ engineers and a partner data-integration specialist take the call when a workflow fails or hangs, with response targets from 15 minutes, by plan.

Challenge

The first decision on a failed run is whether to recover it, and Informatica's documentation makes it depend on the status. A Failed workflow cannot be recovered, while Suspended, Stopped, Aborted and Terminated ones can, the last three only if the workflow is enabled for recovery. Recovering from a checkpoint needs repeatable data, and a restarted Email task might send duplicate email. A wrong guess at 02:00 loads rows twice or leaves the warehouse short.

Environment

A PowerCenter domain with an Integration Service and a Repository Service, a nightly batch of workflows with dependencies, a scheduler that starts them with pmcmd, session and workflow logs on the node, and an on-call rota that changes weekly.

Approach

The on-call engineer joins a live bridge, reads the workflow status in Workflow Monitor or with pmcmd getworkflowdetails, then the session and workflow logs. Before anything is recovered or rerun, the engineer checks what the targets already hold. If the cause is a product defect, AceMQ prepares the logs, versions and a reproduction for the customer's team to open the case with Informatica.

Solution

  1. Read the workflow and task status and the session log to separate a data error from a service or node problem
  2. Check whether the Integration Service answers a ping with pmcmd pingservice, and why a workflow shows Unknown Status or sits in Preparing to Run or Waiting
  3. Choose recover, restart or rerun for each failed task using the status table and the session's recovery strategy
  4. Check the targets for committed rows before any recovery so the fix does not load the same rows twice
  5. Prepare the evidence for an Informatica case when the cause is a product defect, for the customer's contacts to open
  6. Write the root-cause analysis for a P1 and add the decision to the runbook so the next failure takes minutes

Outcome

The engagement delivers a live engineer on each incident, a written recovery decision with the target check behind it, a root-cause analysis for every P1, and a runbook that grows with each incident. Informatica still supplies patches and product fixes; AceMQ works the operations around them.

Technologies

Informatica PowerCenterIntegration ServiceWorkflow Monitorpmcmd

Need Expert Informatica PowerCenter Support?

AceMQ's senior Informatica PowerCenter engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.