Back to all use cases

Managed ServicesSoftware / Digital Platforms

A data team keeps writing jobs while someone else keeps the Spark platform healthy

Data Platform Team

A typical Tier 3 engagement: AceMQ takes over the Spark operator, executor pod capacity, shuffle storage, history server, upgrades and on-call for Spark on the customer's own Kubernetes.

Overview

A typical request: a data engineering team runs Apache Spark on Kubernetes in its own cloud account, and the platform work around the jobs has become a second job. Under Tier 3 managed services, AceMQ operates the Spark layer inside the customer's own cluster and account, delivered by AceMQ with partner Spark specialists under one contract. It is a scope-and-process engagement, not a record of past results.

Challenge

Spark on Kubernetes ships defaults that suit a tutorial. Scratch directories are emptyDir volumes that do not outlive the pod, event logging is off until someone turns it on, dynamic allocation needs shuffle tracking or decommissioning to be set deliberately, and the container memory limit sits above the JVM heap. Teams meet these one incident at a time, usually at night, and the knowledge sits with whoever fixed it last.

Environment

Apache Spark 3.5 or 4.x on a Kubernetes cluster in the customer's AWS, Azure or Google Cloud account, submitted through spark-submit or the Apache Spark Kubernetes Operator, with object storage for data and Prometheus for monitoring.

Approach

AceMQ reads the cluster, the submit path and the job calendar with the team's engineers, then writes down the platform decisions: namespaces and quotas, shuffle and scratch storage, the scaling arrangement, the history server and event log location, and the upgrade path for Spark, the JVM and Python. After a rehearsal against copies of real jobs, AceMQ agrees access and a change process in the customer's account and takes the pager.

Solution

  1. Namespace, quota, service account and pod template review, with the configuration moved into the customer's repository
  2. Shuffle and scratch storage chosen and sized: local volumes, persistent volume claims, shuffle tracking or decommissioning
  3. Event logging switched on to shared storage with retention, and a history server that survives restarts
  4. Monitoring for pending pods, executor loss, driver restarts, retried stages and scratch volume fill, routed to the AceMQ on-call
  5. Upgrade plan for Spark, the JVM, Scala and Python read against the project's migration guides and rehearsed on copies of the customer's jobs
  6. Takeover of the pager with response targets from 15 minutes, by plan, and a written root-cause analysis after each P1

Outcome

The customer receives a documented platform configuration in its own repository, a tested upgrade path, monitoring and runbooks, and a handover pack. The platform, the upgrades and the pager move to AceMQ, and the data team returns to writing and tuning jobs. The cluster, the data and the cloud bill stay in the customer's account.

Technologies

Apache SparkKubernetesPrometheus

Looking for Apache Spark Managed Operations?

AceMQ's senior Apache Spark engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.