01
Executor and Memory Sizing
Cores and memory per executor, memory overhead, the unified memory split between execution and storage, and what each job requests against what it uses.
Spark Health Check
Apache Spark Tuning: Executors, Shuffle, Skew and Cost Reviewed
Last reviewed by Tyler Eastridge
Spark performance tuning means fitting memory, partitions, shuffle and join strategy to your data and cluster. A Spark health check is an engineer-led review of how your jobs really run: executor and memory sizing, shuffle and spill, partition skew, adaptive query execution, file and partition layout, and what the cluster costs for the work it does. You get a written report, a ranked tuning list and the evidence behind each finding, on Spark running on YARN, Kubernetes, EMR, Dataproc, Databricks or standalone. It is delivered by AceMQ with partner Spark specialists.
- areas reviewed on every estate
- 6
- weeks from kickoff to walkthrough
- 1-4
- hour to answer a quote request
- 1
Is This You?
You probably need this if…
- 01
Executors die with out-of-memory errors and the standing fix has been to raise executor memory again
- 02
One task in a stage runs for an hour while the other 199 finish in a minute
- 03
Nightly jobs creep towards the deadline for the reports that depend on them
- 04
Stage detail in the Spark UI shows gigabytes of shuffle spill to disk
- 05
Nobody can say how many cores and how much memory each job asks for, or how much it uses
- 06
A job that worked for two years fails after a lookup table grew past the broadcast threshold
- 07
Cloud spend on Spark clusters rises while the data volume does not
Outcomes
Where you are now, and where you end up
Concrete state changes, not deliverable counts. This is what actually differs about your Apache Spark estate when the engagement closes.
Before
Executor and memory settings were copied from a blog post or a previous job.
After
Requested cores and memory compared with peak usage per job from event logs, with the over-provisioned and under-provisioned jobs listed.
Before
Slow stages are blamed on the cluster.
After
Stage-level evidence of where the time goes: skewed keys, spill, a full shuffle that was not needed, or too many small tasks. Each finding names the fix.
Before
Adaptive query execution is assumed to be handling it.
After
A check of which AQE behaviours are on, which thresholds suit your data, and where a hint or a hand-set partition count overrides them.
Before
The bill goes up and nobody owns it.
After
Idle and oversized compute identified per job and per cluster, ranked by effort against saving, with the changes safe to make first.
Scope
What's covered
- 02
Shuffle, Spill and Skew
Shuffle read and write volume and spill per stage, partition counts, skewed join and aggregation keys, and unnecessary repartitions.
- 03
Adaptive Query Execution
Whether AQE is on, how it coalesces and splits shuffle partitions on your data, the skew-join thresholds, and which hints or settings override it.
- 04
Partitioning and File Layout
Input partition sizing, small-file patterns, table partitioning, and the effect of file counts on task counts and scheduling overhead.
- 05
Cost and Cluster Use
Idle executors, dynamic allocation, cluster shape and autoscaling, and jobs better served by a smaller or different engine.
- 06
Versions and Platform Fit
Spark release against the project's maintenance policy, platform-specific settings on YARN, Kubernetes and Databricks, and what blocks an upgrade.
The Engagement
How it actually runs
Every phase has a defined duration and a concrete artifact handed over at the end of it. You always know what stage you're in and what you've received.
- Phase 11 day
Intake and Scoping
We agree which clusters, jobs and platforms are in scope, the Spark and platform versions, the deadline each job has to meet, and what your engineers can export. Delivery is remote and guided by your engineers over screen share. We ask first for the Spark and Hadoop versions or distribution, where it runs, cores per node, and the current support window.
You receive
- Scope, job list and focus areas
- Export checklist: event logs, Spark configuration, scheduler history and cluster definitions
- Known incidents and deadlines collected
- Phase 21-4 weeks, by scope
Review
A senior engineer and, where the estate needs it, a partner Spark specialist work through the six areas against your event logs and run history, with your team in the room where a decision needs context. The duration depends on the number of jobs, the focus areas you choose and what turns up.
You receive
- Per-job profile of runtime, shuffle, spill and resource use
- Findings with evidence attached
- Notes on anything that blocks a version upgrade
- Phase 3Within a week of the review
Written Findings
The report, ranked by what costs or fails first. Every finding carries the change, the effort, and the risk of leaving it. We say which changes are configuration, which need code, and which depend on the data.
You receive
- Written report
- Ranked tuning list with the setting or code change per item
- Standards for sizing new jobs
- Phase 4At the end of the review
Walkthrough and Plan
A walkthrough with the engineer who did the work, then a sequenced plan your team can run or hand to us. Remediation is scoped separately, so the report is useful whoever makes the changes.
You receive
- Findings walkthrough
- Sequenced plan: change now, change next, leave alone
What a Spark health check reads, and where the evidence comes from
Spark records most of what a review needs. Event logs and the history server hold stage, task and executor detail for finished applications, and the SQL tab shows the physical plan. A finding should trace to a number your engineers can open, not to anyone's recollection of the job.
| Area | Evidence we read | What it tells us |
|---|---|---|
| Executor and memory sizing | Executor tab, peak memory metrics, spark.executor.* and spark.memory.* settings, container or pod limits | Requested versus used memory and cores, and whether failures are heap, overhead or container limits. |
| Shuffle, spill and skew | Stage detail: shuffle read and write size, spill to memory and disk, task duration and input size distribution | Which few stages carry the runtime, and whether one key or one partition holds the data. |
| Adaptive query execution | SQL tab plans, AQE settings, join hints in code | Whether the plan changed at runtime, and where a hint or fixed partition count prevents it. |
| Partitioning and file layout | Input partition counts, file counts and sizes, spark.sql.shuffle.partitions, repartition and coalesce calls | Too many tiny tasks or too few large ones, and avoidable full shuffles. |
| Cost and cluster use | Scheduler history, cluster definitions, executor add and remove events, idle time | Idle executors, oversized clusters, and jobs that wait on a few long tasks. |
| Versions and platform fit | Spark and Hadoop versions, platform release notes, deprecated settings in use | Where an upgrade removes a workaround, and what would break. |
Spark tuning starts from the documented defaults
Many tuning settings are inherited from an older cluster or a tutorial. The review compares what is set with what Spark documents as the default, and with what the job needs. These are the ones that matter most, from the Spark documentation checked on 31 October 2026.
| Setting | Spark default | What we look for |
|---|---|---|
| spark.sql.shuffle.partitions | 200 | A fixed count that no longer matches the data: too many tiny tasks on small stages, or huge partitions that spill on large ones. |
| spark.sql.adaptive.enabled | true (default since Spark 3.2.0) | AQE switched off by an old setting, or defeated by hard-coded partition counts and hints. |
| spark.sql.adaptive.advisoryPartitionSizeInBytes | 64 MB | Whether the target suits your data. It applies when AQE coalesces small shuffle partitions or splits skewed ones. |
| spark.sql.adaptive.coalescePartitions.parallelismFirst | true | With the default, AQE ignores the 64 MB target and only respects a 1 MB minimum. The Spark documentation recommends false on a busy cluster. |
| spark.sql.adaptive.skewJoin thresholds | Enabled; a partition is skewed above 5 times the median and 256 MB | Skew that falls just under the thresholds, and skew in aggregations, which skew-join handling does not cover. |
| spark.sql.autoBroadcastJoinThreshold | 10 MB | A threshold raised years ago, and BROADCAST hints in code, which Spark honours even when statistics are above the threshold. |
| spark.sql.files.maxPartitionBytes | 128 MB | Input partition sizes that give too many tasks over small files, or too few over large ones. |
| spark.executor.memoryOverhead | 10% of executor memory, minimum 384 MiB (40% for Kubernetes non-JVM jobs) | Executors killed by the container limit rather than the JVM heap, which is common with PySpark and native libraries. |
| spark.memory.fraction | 0.6 of heap minus 300 MB | A changed value, since the Spark documentation recommends leaving it at the default. |
| spark.dynamicAllocation.enabled | false | Idle executors held for the whole job, and whether shuffle tracking, an external shuffle service or decommissioning is configured if it is turned on. |
| spark.local.dir | /tmp | Shuffle and spill files on a slow or small disk. |
A common finding
Raising executor memory hides a skewed key. The job fails later, costs more per run, and the stage still waits on its slowest task. Skew and spill evidence comes before any change to memory settings.
Executor sizing, memory and spill
An executor needs memory for the JVM heap, for off-heap and native overhead, and for the work in each of its task slots. When a stage spills, it ran out of execution memory and wrote intermediate data to disk. When a container is killed, the overhead was too small.
- Heap or overhead — A Java out-of-memory error points at the heap. A container or pod killed by the platform with no Java error points at overhead, and in PySpark that is usually Python worker memory.
- Cores against memory — Each core in an executor runs a task at the same time, so more cores per executor means less memory per task. We read per-task memory against the biggest partitions rather than apply a ratio.
- Execution and storage — Cached data and shuffle work share a memory region. Cached tables that are read once, or never unpersisted, take memory from the shuffle.
- Caching and serialization — Cache only data that is reused, and unpersist it. Serializer settings matter mainly for RDD-based jobs, since DataFrame and SQL jobs use Spark's own internal encoding. Columnar formats such as Parquet and sensible compression reduce what is read at all.
- Spill — Spill to disk is slower than memory but not always a fault. We look at which stages spill, how much, and whether fixing skew or partition size removes it.
Shuffle, skew and adaptive query execution
Skew is the usual reason one task runs for an hour while the rest finish in a minute. A join or group on a column where a null placeholder or default value holds a large share of the rows sends all of them to one partition. Adaptive query execution can split skewed partitions in a sort-merge join at runtime, which fixes many cases without code changes.
It does not fix all of them. Skew in an aggregation is not handled by skew-join splitting. A hard-coded repartition can remove the chance for AQE to coalesce. A broadcast hint forces a broadcast even when the table has grown, so a job that worked for two years fails when a lookup table outgrows the driver's memory. We read the physical plan to see what Spark actually did, then choose between enabling or tuning AQE, salting a hot key, removing a null-key path, or deleting a hint.
Cost, cluster use and Spark versions
Spark cost is mostly idle and oversized compute. Dynamic allocation is off by default and, when turned on, needs shuffle tracking, an external shuffle service or decommissioning so executors can be released without losing shuffle data, and executors holding cached data are never removed by default. A cluster sized for one heavy job at 2 a.m. can sit idle for the rest of the day. For Databricks spend measured by DBU, the cost-optimization page goes further.
Version matters for tuning because newer releases remove workarounds. As of 31 October 2026 the Apache Spark project lists 4.3.0 as released on 30 October 2026. Its versioning policy maintains non-LTS branches with bug fixes for 6 months and LTS branches for 18 months, and the 3.5 line has an extended period of security fixes only that ends in November 2027. The health check records which line you run, what your platform supports, and what stands between you and a supported release.
Who this is not for
- Application rewrites — The report tells your developers what to change in code. We do not rewrite your transformation logic as part of a health check.
- Promised savings — We do not quote a percentage by which a job will speed up or a bill will fall before we have seen the logs.
- A single failing job tonight — If a production job is down now, that is an incident. Use Spark support for an emergency response.
- Data science tuning — Model training and notebook performance are outside this review. We look at the Spark jobs that feed them.
Typical engagements
How these engagements run
Typical engagement scenarios: what the work covers and what it delivers. They describe the service, not past results.
Databricks DBU Cost Governance
Global Payments Processor
Right-sizing Databricks compute by moving scheduled work off all-purpose clusters and tightening autoscaling, instance selection, and idle timeouts.
Databricks
Apache SparkDelta Lake
TerraformDatabricks Unity Catalog Migration
Healthcare Analytics Provider
Migrating off the legacy Hive metastore to Unity Catalog with external location mapping, table upgrades, and a grant model that survives audit.
DatabricksDelta LakeApache SparkTerraformDatabricks Delta Small-File Remediation
Digital Retail Platform
Fixing Delta tables where streaming writes and over-partitioning have produced millions of tiny files, stalling reads and vacuum operations.
DatabricksDelta LakeApache SparkDatabricks Job Cluster Failure Support
Insurance Services Group
Named-engineer support for production Databricks job failures — driver OOM, spot reclamation, library conflicts, and workflow retry storms.
DatabricksApache SparkDelta LakeApache Spark Executor OOM and Partition Skew Remediation
Telecommunications Operator
Fixing nightly jobs where a handful of skewed keys concentrate data onto a few executors and drive repeated out-of-memory failures.
Apache Spark
Apache HadoopDelta LakeYARNApache Spark Broadcast Join Failure Support
Industrial Automation Manufacturer
Resolving jobs that fail after a dimension table grows past the broadcast threshold and the optimizer keeps trying to broadcast it anyway.
Apache SparkDelta Lake
Apache HiveApache Spark Shuffle and Spill Tuning
National Retail Bank
Reducing shuffle write volume and disk spill on nightly batch jobs so the processing window fits inside the reporting deadline.
Apache SparkApache HadoopYARNApache HiveApache Spark Workload and Cost Assessment
Media Streaming Provider
Profiling a Spark estate to find over-provisioned jobs, redundant pipelines, and workloads better served by something other than Spark.
Apache SparkDatabricksDelta Lake
Kubernetes
FAQ
Questions about Apache Spark health check
What does an Apache Spark tuning engagement cover?
Executor and memory sizing, shuffle and spill, partition skew, adaptive query execution, file and partition layout, cluster cost and Spark version. It starts from your event logs and Spark configuration and ends in a written, ranked tuning list with the evidence for each item.
How do I fix Spark executor out-of-memory errors?
Find out which kind they are first. A Java heap error and a container killed for exceeding its memory limit have different causes, and both are often a symptom of a skewed partition. Check the stage's task input sizes before raising executor memory, which delays the failure and raises cost.
Is adaptive query execution on by default?
Yes. It has been enabled by default since Apache Spark 3.2.0. It coalesces small shuffle partitions, switches join strategy at runtime and splits skewed partitions in sort-merge joins, but a hard-coded partition count or a broadcast hint can limit what it does.
Which clusters and platforms can you review?
Spark on YARN, Kubernetes, EMR, Dataproc, Databricks and standalone clusters. Delivery is remote and guided by your engineers, who export event logs and configuration. We do not need standing access to your clusters.
How long does a Spark health check take?
One to four weeks from kickoff to walkthrough, depending on the number of jobs, the focus areas and what we find. The scope is agreed on the first day, and you get a written quote within 1 hour of asking.
Let's talk
Talk to an Engineer About Your Spark Estate
Tell us which Spark version and platform you run and which jobs hurt. We will scope a health check and reply to your quote request within 1 hour.
Get in Touch
Talk to an Apache Spark Expert
Send us a message and we'll follow up within 1 hour — or book a free 30-min consultation directly.
305-204-2607info@acemq.com66 W. Flagler St. 9th Floor
Miami, FL 33130
Miami, FL 33130
Prefer to talk now? Call us directly or use the consultation tab to find a time that works.