Move off the Hive metastore without breaking every job in the workspace
Apache SparkTerraformCustomers end up with a single governed catalog across workspaces, access decisions expressed as group grants instead of cluster configuration, and lineage available for audit. Teams migrate domain by…
Overview
Unity Catalog replaces per-workspace Hive metastores with an account-level catalog that carries lineage, row and column controls, and a single grant model. The migration is straightforward in principle and messy in practice, because years of tables, mount points, and hard-coded paths have to be reconciled first.
Challenge
Legacy workspaces typically have hundreds of tables registered against DBFS mounts, a mix of managed and external tables with unclear ownership, and notebooks that reference two-level names that Unity Catalog's three-level namespace will not accept. Access control lives in cluster-scoped ACLs and instance profiles that auditors cannot easily read. Cutting over all at once is not an option when the same tables feed regulated reporting.
Environment
Databricks workspaces on AWS or Azure with an existing Hive metastore, DBFS mounts, and mixed managed and external Delta tables.
Approach
AceMQ inventories the existing metastore, classifies every table by storage location and ownership, and maps cloud storage paths to Unity Catalog external locations and storage credentials. Tables move in waves by domain, running in parallel against both metastores until consumers are cut over. Grants are rebuilt from group membership rather than replicated table by table, so the resulting model is reviewable.
Solution
- 1Full metastore inventory classifying managed versus external tables, storage paths, and active consumers
- 2External location and storage credential design mapped to existing cloud IAM boundaries
- 3Wave-based table upgrade with dual-registration so consumers migrate on their own schedule
- 4Notebook and job scanning to find two-level table references and DBFS mount paths that need rewriting
- 5Group-based grant model with catalog, schema, and table privileges defined in Terraform
- 6Lineage and audit log validation against the compliance evidence the team actually has to produce
Outcome
Customers end up with a single governed catalog across workspaces, access decisions expressed as group grants instead of cluster configuration, and lineage available for audit. Teams migrate domain by domain rather than absorbing one high-risk cutover weekend.
Technologies
Apache SparkTerraformRelated Use Cases
Databricks DBU Cost Governance
Right-sizing Databricks compute by moving scheduled work off all-purpose clusters and tightening autoscaling, instance selection, and idle timeouts.
Databricks Delta Small-File Remediation
Fixing Delta tables where streaming writes and over-partitioning have produced millions of tiny files, stalling reads and vacuum operations.
Need Databricks Architecture Guidance?
AceMQ's senior Databricks engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.