Overview
Spark on Kubernetes adds a container limit on top of the JVM heap, and moves shuffle data onto pod storage. Settings copied from a YARN cluster or a tutorial often fit neither. AceMQ reviews the jobs against Spark's documented behaviour and the cluster's real limits, and returns a ranked list of changes.
Challenge
Executor pods are killed by the platform with no Java error in the log, and the standing response has been to raise executor memory. Nightly jobs hold their full executor count while most of it sits idle. Shuffle files are written to a small local volume that fills on the largest stages. Adaptive query execution is assumed to be on, and no one has checked what overrides it. A PySpark job and a JVM job use the same overhead setting.
Environment
Apache Spark on Kubernetes, with batch jobs in Scala and PySpark reading Parquet and Delta tables from object storage and monitored through Prometheus.
Approach
AceMQ reviews event logs, pod specifications, Spark configuration and executor metrics with the team's engineers. It separates heap errors from container kills, compares requested cores and memory with measured peaks, and checks how executors are added and removed. Shuffle volume, spill and partition skew are read stage by stage before any memory setting is changed. A partner Spark specialist joins where the findings call for it.
Solution
- Separation of Java heap failures from container limit kills, with the memory overhead setting reviewed for PySpark and native libraries
- Cores and memory per executor compared with measured peaks, job by job
- Dynamic allocation review: whether it is on, and which of shuffle tracking, decommissioning or an external shuffle service supports it
- Shuffle and spill location, volume sizing and local disk layout for the largest stages
- Adaptive query execution settings, skew-join thresholds and the hints or fixed partition counts that override them
- Input partition sizing and small-file patterns in the source tables
- Spark release against the project's maintenance policy, and what blocks an upgrade
Outcome
The engagement delivers a written report with a per-job profile, a ranked list of configuration and code changes with the evidence for each, and standards for sizing new jobs on the cluster. The risk removed is tuning by guesswork: each change is tied to a measurement, and the order of changes is chosen so the cheap, safe ones come first.
Technologies
Apache Spark
Kubernetes
Prometheus
Databricks
Apache Hadoop
Grafana
Terraform
