When 199 tasks finish in a minute and one runs for an hour, the problem is skew

Overview

Executor OOM in Spark is usually a symptom, and the underlying cause is often that a small number of join or group keys hold a disproportionate share of the rows. AceMQ diagnoses the skew directly rather than escalating executor memory until the job stops crashing.

Challenge

The common pattern is a join on a column where a null placeholder or a default tenant ID accounts for a large fraction of the rows. Those rows all hash to one partition, one executor tries to hold them, and it dies. Teams respond by raising spark.executor.memory, which delays the failure and increases cluster cost without addressing anything. Meanwhile the stage's runtime is bounded by its slowest task, so the job window keeps growing.

Environment

Apache Spark on YARN, Kubernetes, or Databricks, running batch ETL against Hive, Delta, or Iceberg tables.

Approach

AceMQ reads the Spark UI stage detail and event logs to compare per-task input size and shuffle read distribution, which makes skew immediately visible. Remediation is chosen to fit the data: adaptive query execution where the version supports it, salting for persistent hot keys, filtering null-key rows out of the join path, or a broadcast where the dimension side is genuinely small.

Solution

• Per-task input and shuffle read distribution analysis from event logs to identify and quantify the skewed keys
• Adaptive Query Execution enablement with skew join and partition coalescing thresholds tuned to the observed distribution
• Key salting for persistent hot keys, with the corresponding aggregation rewrite
• Null and sentinel key handling removed from the join path where those rows carry no business meaning
• Executor memory, core count, and memory-overhead sizing corrected once the distribution is fixed rather than before
• Partition count and repartition placement review to stop unnecessary full shuffles

Outcome

Technologies