Overview
A typical request: a data engineering team runs Apache Spark on Kubernetes in its own cloud account, and the platform work around the jobs has become a second job. Under Tier 3 managed services, AceMQ operates the Spark layer inside the customer's own cluster and account, delivered by AceMQ with partner Spark specialists under one contract. It is a scope-and-process engagement, not a record of past results.
Challenge
Spark on Kubernetes ships defaults that suit a tutorial. Scratch directories are emptyDir volumes that do not outlive the pod, event logging is off until someone turns it on, dynamic allocation needs shuffle tracking or decommissioning to be set deliberately, and the container memory limit sits above the JVM heap. Teams meet these one incident at a time, usually at night, and the knowledge sits with whoever fixed it last.
Environment
Apache Spark 3.5 or 4.x on a Kubernetes cluster in the customer's AWS, Azure or Google Cloud account, submitted through spark-submit or the Apache Spark Kubernetes Operator, with object storage for data and Prometheus for monitoring.
Approach
AceMQ reads the cluster, the submit path and the job calendar with the team's engineers, then writes down the platform decisions: namespaces and quotas, shuffle and scratch storage, the scaling arrangement, the history server and event log location, and the upgrade path for Spark, the JVM and Python. After a rehearsal against copies of real jobs, AceMQ agrees access and a change process in the customer's account and takes the pager.
Solution
- Namespace, quota, service account and pod template review, with the configuration moved into the customer's repository
- Shuffle and scratch storage chosen and sized: local volumes, persistent volume claims, shuffle tracking or decommissioning
- Event logging switched on to shared storage with retention, and a history server that survives restarts
- Monitoring for pending pods, executor loss, driver restarts, retried stages and scratch volume fill, routed to the AceMQ on-call
- Upgrade plan for Spark, the JVM, Scala and Python read against the project's migration guides and rehearsed on copies of the customer's jobs
- Takeover of the pager with response targets from 15 minutes, by plan, and a written root-cause analysis after each P1
Outcome
The customer receives a documented platform configuration in its own repository, a tested upgrade path, monitoring and runbooks, and a handover pack. The platform, the upgrades and the pager move to AceMQ, and the data team returns to writing and tuning jobs. The cluster, the data and the cloud bill stay in the customer's account.