Move off the Hive metastore without breaking every job in the workspace

Overview

Unity Catalog replaces per-workspace Hive metastores with an account-level catalog that carries lineage, row and column controls, and a single grant model. The migration is straightforward in principle and messy in practice, because years of tables, mount points, and hard-coded paths have to be reconciled first.

Challenge

Legacy workspaces typically have hundreds of tables registered against DBFS mounts, a mix of managed and external tables with unclear ownership, and notebooks that reference two-level names that Unity Catalog's three-level namespace will not accept. Access control lives in cluster-scoped ACLs and instance profiles that auditors cannot easily read. Cutting over all at once is not an option when the same tables feed regulated reporting.

Environment

Databricks workspaces on AWS or Azure with an existing Hive metastore, DBFS mounts, and mixed managed and external Delta tables.

Approach

AceMQ inventories the existing metastore, classifies every table by storage location and ownership, and maps cloud storage paths to Unity Catalog external locations and storage credentials. Tables move in waves by domain, running in parallel against both metastores until consumers are cut over. Grants are rebuilt from group membership rather than replicated table by table, so the resulting model is reviewable.

Solution

• Full metastore inventory classifying managed versus external tables, storage paths, and active consumers
• External location and storage credential design mapped to existing cloud IAM boundaries
• Wave-based table upgrade with dual-registration so consumers migrate on their own schedule
• Notebook and job scanning to find two-level table references and DBFS mount paths that need rewriting
• Group-based grant model with catalog, schema, and table privileges defined in Terraform
• Lineage and audit log validation against the compliance evidence the team actually has to produce

Outcome

Technologies