Find out which Spark jobs earn their compute and which ones do not

Overview

Spark estates grow by accretion. Jobs get copied, resource requests get inflated to make a deadline once and never revised, and pipelines outlive the reports they fed. AceMQ profiles the whole estate and returns a ranked list of what to fix, what to consolidate, and what to delete.

Challenge

Nobody has a complete picture of which jobs run, what they cost, and who consumes their output. Executor counts and memory settings were chosen defensively rather than measured. Several pipelines compute nearly the same aggregate from the same source. A meaningful fraction of jobs process small enough volumes that Spark's overhead exceeds the work, and would run faster as a single-node process.

Environment

Apache Spark on Kubernetes, EMR, Dataproc, or Databricks, with jobs orchestrated by Airflow or a native scheduler.

Approach

AceMQ collects event logs and scheduler history across the estate to build per-job runtime, resource allocation, and utilization profiles. Allocated resources are compared against peak observed usage to find over-provisioning, and output lineage is traced to find pipelines whose consumers no longer exist. Each finding carries a specific recommended action rather than a general observation.

Solution

• Estate-wide job inventory with runtime, allocated resources, peak utilization, and cost per run
• Over-provisioning analysis comparing requested executor memory and cores against measured peak usage
• Duplicate and near-duplicate pipeline detection through source and output lineage
• Identification of small-volume jobs where Spark overhead exceeds the work and a single-node engine is faster and cheaper
• Orphaned pipeline detection where downstream consumers no longer read the output
• Prioritized remediation backlog ordered by cost saved against engineering effort required

Outcome

Technologies