Overview
A typical request: a platform team that already runs Kubernetes wants to know how much of its Databricks job estate could run as open-source Spark on that cluster, writing Apache Iceberg tables, and how much depends on features that only Databricks provides. AceMQ delivers a written assessment, scope and process, not a migration.
Challenge
A Databricks workspace accumulates jobs that look alike from the bill but differ in what they need. Some are plain Spark transformations. Others use Photon, serverless compute, Delta Live Tables, Unity Catalog permissions or notebooks that other teams depend on. Without an inventory, the proposal to run Spark on Kubernetes is either rejected as too risky or approved for the wrong jobs.
Environment
Databricks on AWS, Azure or GCP with scheduled jobs and Delta tables, and a team that operates Kubernetes and wants to evaluate Spark and Iceberg there.
Approach
AceMQ inventories jobs and their runtime, libraries, cluster settings, table formats and downstream consumers, then sorts each into candidate to offload, candidate to keep and needs a decision. For the candidates it sets out the target design: Spark on Kubernetes, Iceberg tables on object storage, a catalog, a scheduler and the table maintenance jobs that Databricks currently does for the team. The assessment tests one candidate job read-and-write path against the chosen catalog before any recommendation is made.
Solution
- Job inventory with runtime, libraries, cluster configuration, tables read and written, and owners
- Classification of each job as offload candidate, keep on Databricks, or decision needed, with the Databricks-only features each one uses
- Target design for Spark on Kubernetes: operator or submit model, executor sizing, autoscaling and shuffle storage
- Iceberg table and catalog design, including who owns compaction and snapshot expiry once Databricks no longer does
- Exit-path test: one Unity Catalog table read and written from an external engine through the Iceberg REST Catalog API, with limits recorded
- Operating model and support tier options for the new platform, and a costed plan for the first workload
Outcome
The team receives a written inventory, a sorted list of jobs with the reasons, a target design for the offload candidates and the recorded results of the exit-path test. The decision to move, keep or defer each workload is made on that evidence, and the team can still choose to stay on Databricks.
Technologies
Databricks
Apache SparkApache Iceberg
Kubernetes
Terraform
Starburst
Prometheus
