Managed Apache Spark
Last reviewed by Tyler Eastridge
AceMQ runs open-source Apache Spark for you: the clusters, the operator or submit path, shuffle storage, the history server, upgrades and the pager. We operate it on your Kubernetes, YARN or standalone clusters in your own account (Tier 3), or on infrastructure AceMQ runs (Tier 4). If Databricks, Amazon EMR or Google's managed Spark service already suits you, those can be the better choice, and the table below says where each approach stops.
Support quotes answered within 1 hour, by a senior engineer.
Planning a build, upgrade or migration instead? See AceMQ Apache Spark consulting and services.
Customers
Partners
What's Included
No add-on pricing for incidents. No per-ticket charges. One contract covers the whole surface.
We run the layer your jobs land on: Spark on Kubernetes through spark-submit or the Apache Spark Kubernetes Operator, YARN queues, or standalone masters and workers. Namespaces, quotas, service accounts and pod templates are kept in code in your repository, and capacity is reviewed against the jobs that actually run.
Spark's scratch directories decide whether a lost node costs a retry or a whole stage. On Kubernetes each local directory is an emptyDir that does not outlive its pod, and Spark's own docs list the external shuffle service as future work there. We choose and run the arrangement: local volumes, persistent volume claims, shuffle tracking or decommissioning, sized and monitored.
Event logging is off by default, and its default directory is a local temporary path. We switch it on, point it at shared storage, run the history server, set retention and keep the logs your engineers debug from. A finished job whose UI has vanished is the first thing people lose, and the last thing they should.
We manage monitoring of what predicts Spark incidents: pods stuck pending, executor loss rate, driver restarts, failed and retried stages, scratch volume fill and queue wait. Then we take the pager, so an alert reaches an engineer who knows your cluster, your job calendar and your storage layout.
Spark, the JVM, Scala and Python move together. We plan each upgrade against the project's own migration guides, rehearse it on copies of your real jobs, write the rollback down and run it in an agreed window. Version upgrades, including Spark 3.5 to 4.x, run as projects inside the service.
Credentials come from your secrets backend, storage access follows least privilege, and who can submit to which queue or namespace is reviewed. We also watch for idle capacity, orphaned pods and runaway retries, and report them. Access to your environment follows your change process and is logged.
Why AceMQ
Run managed Spark in your own account, cluster or data centre (Tier 3) when your data and credentials must stay inside your boundary, or have AceMQ host it (Tier 4) when you want working capacity and no infrastructure to run. Either way the operator is a Spark specialist under an AceMQ contract.
AceMQ does not resell Databricks, EMR or Google's service. When one fits better than we do, we say so, and we support the jobs on them at Tier 1 or 2.
The same contract can cover the Spark clusters and the Kafka, Airflow or databases that feed them, so a late pipeline has one owner on the bridge. AceMQ brings in partner Spark specialists as the work needs them. Response targets from 15 minutes, by plan, are response times, not resolution times, and every P1 ends with a written root-cause analysis.
Cluster recovery, stuck-job and upgrade runbooks, deployment code and monitoring configuration are maintained as yours. If you move between your infrastructure and ours, or take operations in-house, a current handover pack is ready.
Cloud, Kubernetes, bare metal, hybrid, and air-gapped — including environments where you can't give us outbound network access.
Compared
Databricks, Amazon EMR and Google's Managed Service for Apache Spark each run Spark inside their own service. AceMQ runs open-source Spark in your account or ours. These are the differences that decide it, from the vendors' own pages as read on 4 November 2026.
| Question | A cloud Spark service | AceMQ at Tier 3 (your infrastructure) | AceMQ at Tier 4 (ours) |
|---|---|---|---|
| Who runs Spark | The provider runs the platform to the edge of its service. Google, for example, offers clusters you size yourself, serverless with no cluster or job resources to manage, and containers on GKE. | AceMQ engineers operate Apache Spark on your Kubernetes, YARN or standalone clusters, with system access. | AceMQ engineers operate Spark on infrastructure AceMQ runs, sized to your workload. |
| Which Spark you get | The provider's own release. Databricks Runtime 18 LTS is built on Spark 4.1.0 and supported to 10 June 2029. The newest EMR 7.x release on AWS's page, 7.14.0, ships Spark 3.5.8. | The Apache release you choose: 4.3.0 (30 October 2026) or any maintained line down to 3.5.x, which has an extended security-only period to November 2027. | The same Apache releases, upgraded to an agreed plan. |
| Where data and credentials sit | It depends on the service and mode you buy. Check where compute and storage run before you commit. | In your account, your network and your object storage. Credentials stay with your secrets backend. | On AceMQ infrastructure. Data handling terms are agreed in writing before onboarding. |
| Who answers when a job fails | The provider supports the platform and its runtime. Job logic, data skew and memory configuration stay with you or your support provider. | AceMQ takes the pager: response targets from 15 minutes, by plan, and a written root-cause analysis after every P1, covering the job and the cluster beneath it. | The same SLA, with the cluster also ours to keep healthy. |
| How you are charged | By the provider's pricing for its service. AceMQ prints no prices for it. | Quote only. Your own infrastructure bill plus an operations fee. | Quote only. A hosted quote that includes the infrastructure. |
| When it fits | A small team, one cloud, and a platform the provider can carry end to end. | Open-source Spark kept for cost, version control or data location, with specialists running the cluster. | Spark capacity without a platform team or infrastructure of your own. |
Sources: spark.apache.org versioning policy and 4.3.0 documentation, docs.databricks.com Databricks Runtime release notes (updated 30 October 2026), docs.aws.amazon.com EMR 7.x release guide, docs.cloud.google.com Managed Service for Apache Spark overview. Vendors change terms, so check them before you commit. Databricks, EMR and Google's service are run by their providers: AceMQ supports the jobs on them at Tiers 1 and 2, and operates open-source Spark itself at Tiers 3 and 4.
The same four tiers cover Kafka, Airflow, PostgreSQL and RabbitMQ; the escalation-first option is set out in managed open-source operations.
On Kubernetes
Spark on Kubernetes is a set of defaults that suit a tutorial and not a shared production cluster. These are the ones that bite, from the Spark 4.3.0 documentation as read on 4 November 2026.
The Apache Spark Kubernetes Operator is a separate Apache project that adds SparkApp and SparkCluster resources. Teams that submit through spark-submit or an orchestrator keep doing so. We run whichever you use, and the choice sits in your repository.
Every spark.local.dir becomes an emptyDir that does not persist beyond the pod. With tmpfs on, it counts toward pod memory. A persistent volume claim or a shuffle plugin such as KubernetesLocalDiskShuffleDataIO is how shuffle data survives a lost executor.
Dynamic allocation is off by default and needs shuffle tracking, shuffle block decommissioning or a reliable shuffle plugin. Shuffle tracking is on by default, but decommissioning, spark.decommission.enabled, is off. We set the combination to match how your nodes are reclaimed.
The overhead factor is 0.10 for JVM jobs and 0.40 for non-JVM jobs such as PySpark. A pod killed for memory with no Java error in the log is often the container limit and not the heap.
Executor pods are requested 20 at a time with a one-second delay by default. On a busy cluster, pending pods and quota limits decide start-up time, so we watch queue wait next to executor counts.
Spark 4.3 sets allowPrivilegeEscalation to false on driver and executor containers, and executors now connect to the driver pod IP and not its Service. Both can break a restrictive network policy or an old pod template, so each upgrade is rehearsed.
Executor sizing and job tuning are a different engagement; the Spark health check covers them. Spark health check
Spark 4
Spark 3.5.x has an extended security-only period that ends in November 2027. Non-LTS lines are maintained for six months, the last release of a major line becomes an LTS for 18 months, and the project says the 4.x LTS will be 4.5.0. These are the things that change an upgrade plan.
Spark 4.3 runs on Java 17, 21 or 25, Scala 2.13 and Python 3.11 or newer. Scala 2.13 applies since 4.0.0, so Scala applications and JARs built for 2.12 need rebuilding.
Support for Apache Mesos was removed in Spark 4.0. Estates still on Mesos need a move to Kubernetes, YARN or standalone first, and we scope that as a project.
Spark 4.3 puts CSRF tokens on UI actions that change state and requires POST requests for non-YARN deployments. Scripts that kill jobs through a plain GET stop working until they fetch a page first, or the old behaviour is switched back on.
On 4 November 2026 the newest release is 4.3.0 (30 October), after 4.2.1, 4.1.4 and 4.0.5 on 27 October. AceMQ covers the current release and the five end-of-life lines before it, so a 3.x estate is run while the move is planned.
Upgrades are tried on copies of your own jobs and data samples, the results diffed against the current version, and the rollback written down before the window opens.
A cloud service ships its own Spark build on its own schedule. Databricks Runtime 18 LTS carries Spark 4.1.0, and the newest EMR 7.x release on AWS's page carries 3.5.8. Open-source Spark on your clusters can move earlier or later by your choice.
Sources: spark.apache.org versioning policy, 4.3.0 documentation and migration guides, docs.databricks.com and docs.aws.amazon.com, read 4 November 2026. Apache Spark support options
Choosing
A cloud provider's Spark service is a good answer for many teams, and AceMQ will say so. These are the situations that decide it.
You run in one cloud, the team is small, and you want the provider to carry the platform. Databricks Runtime LTS releases have printed end-of-support dates, and a provider's serverless mode removes cluster work altogether.
You keep open-source Spark for cost, want to pick your own version and upgrade date, need it on-premises or air-gapped, or run Spark on Kubernetes you already operate. One operator can also cover Spark, Kafka, Airflow and the databases beside it.
You are happy on Databricks, EMR or Google's service but want a senior engineer for the job layer: skew, spill, memory errors and missed windows. That is Tier 1 or 2 on the support page, not a managed contract.
Moving off one means rebuilding job submission, storage access and monitoring, replacing provider-specific features and running old and new side by side until outputs match. We scope that as a project and then run the platform.
Not sure which? An assessment compares them for your job mix, data location and team. Apache Spark consulting
Ownership
A failed Spark job can be the platform or the code. The contract says which is which, and the price is built on the same split.
Clusters, operator or submit path, shuffle storage, the history server, upgrades, monitoring and the pager.
What a job computes, its schedule, its data and its tests. Your engineers write and merge it through your pipeline.
When a job fails on skew, spills to disk or is killed for memory, we find the cause and tell you which stage and why. Fixing the job is yours, or a consulting engagement.
Clusters and environments, job volume, Kubernetes, YARN or standalone, streaming or batch, cover hours from business hours to 24/7, and Tier 3 or Tier 4. At Tier 3 you pay your cloud for infrastructure and AceMQ for operations; at Tier 4 the infrastructure is in the quote. Quotes come back within 1 hour of your request.
Not a fit for
Some Spark estates are better left with a provider, or with nobody on call at all.
If the need is a cluster created from a console for an afternoon's job and paid by the hour, a cloud Spark service or its serverless mode does that, and a managed contract adds nothing.
If your pipelines depend on Databricks features such as Unity Catalog or Delta Live Tables, moving to open-source Spark is a different project. We can assess it, and we support jobs on Databricks at Tier 1 or 2.
A managed platform keeps Spark healthy. It does not turn skewed or wasteful jobs into good ones. That is consulting work, and we scope it separately.
A handful of batch jobs with no uptime target do not need a managed Spark contract. A supported version and an event log is enough.
Running several engines? One contract can also cover Kafka, PostgreSQL, Redis and more. Data engineering consulting
Escalation Path
Every tier reaches a senior Apache Spark engineer. There is no tier-1 triage layer to get through.
Response targets by plan; your contract sets the tier.
P1 escalation path
A replay of one P1, from the first page to the written root-cause analysis
Response targets from 15 minutes, by plan; 24/7.
A response time, not a resolution time.
P1 opened, on-call engineer paged
Phone, email or Slack reaches the on-call senior engineer directly, whether the incident is in RabbitMQ, Kafka, Redis, PostgreSQL or Kubernetes. No web form, no tier-1 queue.
Senior engineer on your bridge
A senior engineer who already knows your environment joins a live bridge, so nobody has to re-explain your topology.
Specialists across the stack join
When an incident crosses components, the engineers who know each one join. For RabbitMQ, the path runs to the core team.
Written root-cause analysis
Root cause, the fix applied and the prevention steps, delivered after every P1.
Typical engagements
Apache Spark support and remediation work from AceMQ. Each card links to the full write-up: the pipelines, what AceMQ diagnosed or changed, and the result.
Global Payments Processor
Right-sizing Databricks compute by moving scheduled work off all-purpose clusters and tightening autoscaling, instance selection, and idle timeouts.
Databricks
Apache SparkDelta Lake
TerraformDigital Retail Platform
Fixing Delta tables where streaming writes and over-partitioning have produced millions of tiny files, stalling reads and vacuum operations.
DatabricksDelta LakeApache SparkInsurance Services Group
Named-engineer support for production Databricks job failures — driver OOM, spot reclamation, library conflicts, and workflow retry storms.
DatabricksApache SparkDelta LakeTelecommunications Operator
Fixing nightly jobs where a handful of skewed keys concentrate data onto a few executors and drive repeated out-of-memory failures.
Apache Spark
Apache HadoopDelta LakeYARNIndustrial Automation Manufacturer
Resolving jobs that fail after a dimension table grows past the broadcast threshold and the optimizer keeps trying to broadcast it anyway.
Apache SparkDelta Lake
Apache HiveMedia Streaming Provider
Profiling a Spark estate to find over-provisioned jobs, redundant pipelines, and workloads better served by something other than Spark.
Apache SparkDatabricksDelta Lake
KubernetesSupport options
AceMQ offers Apache Spark support in four tiers, each handing more of the day-to-day work to AceMQ: 24/7 support and incident management for pipelines your team runs, proactive monitoring that finds skew, spill and small-file problems first, managed Spark on your own Kubernetes, YARN or standalone clusters, and fully managed Spark on AceMQ infrastructure. Databricks, EMR and Dataproc jobs are covered too. Version cover is the current Spark release plus the five end-of-life release lines before it.
You run it, we back you up
Best for: Data engineering teams that run Spark jobs themselves and need senior cover when a pipeline fails at 3am.
You run it, we watch it with you
Best for: Teams that want the creeping runtime or the missing watermark found in a review, not during a missed SLA.
We run it in your cloud or data centre
Best for: Estates that keep open-source Spark for cost, version control or data location and want the cluster run by specialists.
We host it and run it, as a service
Best for: Teams that want Spark capacity without a platform team to run it.
| Who does what | Tier 1 24/7 support | Tier 2 Proactive monitoring | Tier 3 Managed, your infrastructure | Tier 4 Fully managed by AceMQ |
|---|---|---|---|---|
| Who runs Apache Spark day to day? | Your team | Your team, with AceMQ reviews | AceMQ | AceMQ |
| Where does it run? | Your infrastructure | Your infrastructure | Your cloud account or data centre | AceMQ infrastructure |
| 24/7 incident response | Response targets from 15 minutes, by plan | Response targets from 15 minutes, by plan | Response targets from 15 minutes, by plan | Response targets from 15 minutes, by plan |
| Health checks and CVE alerts | On request | Scheduled | Included | Included |
| Patching and upgrades | Your team, AceMQ advises | Planned with AceMQ | Done by AceMQ | Done by AceMQ |
| Monitoring and on-call | Your team | Your team | AceMQ | AceMQ |
| Data, keys and accounts | Yours | Yours | Yours | Your data, hosted by AceMQ |
Every Tier can start as an ad hoc engagement or an annual partnership, and you can move up or down a Tier as your team changes.
One partner for Apache Spark
Support is one part of what AceMQ does for Apache Spark. The same engineers deliver consulting, health checks, migrations and end-of-life cover, and support the systems that usually run next to it.
Those services run Spark inside their own platform, on their release schedule. AceMQ operates open-source Apache Spark on your clusters or ours with system access, so you choose the version and the upgrade date, and it takes the pager with response targets from 15 minutes, by plan. Both options leave job code with you.
Yes. Tier 3 runs Spark through spark-submit or the Apache Spark Kubernetes Operator on your Kubernetes in your AWS, Azure or Google Cloud account or in your data centre. We also run YARN and standalone clusters. Tier 4 is the alternative when you would rather not run any infrastructure.
We read the project's migration guides against your estate: Java and Scala versions, Mesos if you still use it, scripts that call the Spark UI and the Kubernetes security defaults. Then we rehearse the upgrade on copies of your own jobs and cut over in an agreed window with a rollback written down. Spark 3.5.x keeps security-only fixes until November 2027.
Yes. AceMQ works on Apache Spark in two ways: ad hoc engagements for a defined piece of work, such as a health check, an upgrade or an incident, or an annual contract that covers 24/7 support and proactive reviews. Many teams start ad hoc and move to an annual contract once the first engagement is done.
Managed Apache Spark is quoted per scope, with no list price: the price depends on the number of environments and nodes, the SLA, whether AceMQ runs Apache Spark on your infrastructure (Tier 3) or on ours (Tier 4), and the access and change process you need. Request a quote and AceMQ replies within 1 hour.
Let's talk
Reactive cover, proactive reviews, or AceMQ running Apache Spark for you on your infrastructure or ours. A senior engineer scopes it with you, and quote requests are answered within 1 hour.
Production incident right now?Email an Apache Spark Architect
Send us a message and we'll follow up within 1 hour — or book a free 30-min consultation directly.
305-204-2607info@acemq.comPrefer to talk now? Call us directly or use the consultation tab to find a time that works.
