Find out where the DBUs actually go, then stop paying for idle compute
Apache SparkDelta LakeTerraformCustomers get a ranked list of the workloads driving the majority of DBU spend and a policy set that keeps new work from repeating the same pattern. Most engagements identify enough idle and oversized…
Overview
Databricks bills on DBUs, and the same notebook that costs a few dollars on a job cluster can cost several times that on an interactive all-purpose cluster left running all day. AceMQ audits actual cluster usage against the workloads that run on it and produces a costed remediation plan.
Challenge
Teams that started in notebooks tend to keep production ETL on shared all-purpose clusters because it is convenient. Autoscaling minimums get set high to avoid cold starts, idle timeouts get disabled so nobody loses their session, and cluster policies are either absent or too permissive to enforce anything. The result is a bill that grows faster than the data volume and no clear attribution back to a team or pipeline.
Environment
Databricks on AWS, Azure, or GCP, with workloads spanning notebooks, scheduled jobs, and Delta Live Tables pipelines.
Approach
AceMQ pulls system tables and cluster event logs to build a per-workload DBU profile, separating interactive exploration from scheduled production work. Each expensive workload is examined for whether the cost is compute selection, runtime, or data layout. We then define cluster policies and instance pool configuration that make the cheap path the default path rather than relying on team discipline.
Solution
- 1DBU attribution by workload, team, and cluster type from system tables and cluster event logs
- 2Migration of scheduled pipelines from all-purpose clusters to job clusters with ephemeral lifecycles
- 3Autoscaling floor, idle timeout, and instance family review including Photon and spot suitability per workload
- 4Cluster policies that constrain node types, worker counts, and DBR versions by workspace and team
- 5Instance pool design to cut cold-start cost without holding warm capacity overnight
- 6Tagging and chargeback model so platform spend maps to the team that generated it
Outcome
Customers get a ranked list of the workloads driving the majority of DBU spend and a policy set that keeps new work from repeating the same pattern. Most engagements identify enough idle and oversized compute to cut platform spend meaningfully without changing a single pipeline's logic.
Technologies
Apache SparkDelta LakeTerraformRelated Use Cases
Databricks Unity Catalog Migration
Migrating off the legacy Hive metastore to Unity Catalog with external location mapping, table upgrades, and a grant model that survives audit.
Apache Spark Workload and Cost Assessment
Profiling a Spark estate to find over-provisioned jobs, redundant pipelines, and workloads better served by something other than Spark.
Ready for a Databricks Health Check?
AceMQ's senior Databricks engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.