The Hadoop to Databricks Migration Guide

Databricks said on 13 August 2026 that it had crossed a $7 billion revenue run-rate, growing more than 80% year over year in its second quarter. That tells you the destination is widely adopted and well funded. It says nothing about what your migration will cost, and the steps below are the part that decides that.

Scope the estate before you scope the project. Hadoop clusters often carry datasets and jobs nobody owns any more, so the first deliverable is a list of what you will not migrate.

Build the inventory from evidence, not documentation. For HDFS, list directories by size and last access, because capacity is not the same as use. For Hive, list databases, tables, storage format (Parquet, ORC, Avro, text, SerDe), partition counts and whether each table is managed or external. For jobs, list every Oozie workflow and coordinator with its actions, schedule and owner. For security, export the Ranger or Sentry policies and the groups they name.

A Cloudera to Databricks move adds three items: Impala workloads and their query patterns, the Oozie schedules, and the authorization layer, which is Sentry on CDH and Ranger on CDP.

Databricks recommends a control map that records each capability, its target equivalent and the evidence of parity. Keep it; it becomes the sign-off checklist at cutover.

Databricks publishes its own mapping for HDFS, Hive, YARN, MapReduce and the ingestion tools. The rows for Impala, Oozie and Ranger are our working mapping, not a one-to-one translation.

Copy with DistCp, which runs as a MapReduce job on the cluster you are leaving:

hadoop distcp -update -direct \
  hdfs://nn1:8020/warehouse/sales \
  s3a://acme-lake/warehouse/sales

-update copies files that are missing or differ. Against an object store it compares modification time and length, not checksums, so run your own checksum or row-count comparison afterwards. -direct skips the temporary-file rename, which is expensive on object stores. The link between your data centre and the cloud sets the schedule, so copy in waves and finish each dataset with one incremental pass at cutover.

Land the files under Unity Catalog external locations, each backed by a storage credential, so access is governed from the first query.

CONVERT TO DELTA converts a Parquet table in place: it lists the files, writes a Delta transaction log and infers the schema from the Parquet footers.

CONVERT TO DELTA parquet.`s3://acme-lake/warehouse/sales`
  PARTITIONED BY (sale_date DATE);

PARTITIONED BY is required when you point at a path holding partitioned data, and optional when you name a table already registered in the metastore. The command handles Parquet, and Iceberg tables whose files are Parquet. Hive tables stored as ORC, Avro or text need a rewrite instead, for example CREATE TABLE ... AS SELECT into a Delta table.

Stop the writers first. Files the Delta log does not track are invisible and VACUUM can delete them, so do not append or update during conversion and send every later write through Delta. If several external tables share one directory, converting one breaks the others until each is converted. NO STATISTICS finishes faster, and Databricks then recommends liquid clustering to reorganize the layout and generate statistics.

Which tool applies depends on where the metastore lives.

• On-premises metastore. The data must be in cloud storage first. Hive metastore federation does not support data in on-premises HDFS or a metastore reached in remote Thrift mode, so it cannot bridge a live on-premises cluster. Recreate the tables over the copied files from your exported DDL.
• SYNC. Upgrades Hive metastore external tables to Unity Catalog external tables. Run the dry run first; it returns DRY_RUN_SUCCESS for each table that can move.
• Federation. Lets Unity Catalog govern tables in an external Hive metastore or AWS Glue while jobs migrate one at a time. Foreign catalogs on external metastores are read-only, and Hive functions, UDFs and JDBC-backed tables are not supported. Treat it as a bridge, not a destination.
• UCX. A Databricks Labs project with assessment, group migration and table migration workflows. It is provided as-is, with no SLA.

SYNC SCHEMA main.sales FROM hive_metastore.sales DRY RUN;

The status codes name the blockers: HIVE_SERDE (SerDe tables are not supported), LOCATION_OVERLAP, MULTIPLE_EXT_LOCATIONS and VIEWS_NOT_SUPPORTED (recreate views from SHOW CREATE TABLE). AS EXTERNAL upgrades managed tables as external ones on Databricks Runtime 13.2 and above, and tables in workspace DBFS root need DEEP CLONE instead. SYNC also writes a table property into each source Delta table's log, so the cluster that runs it needs write access to the table location.

Oozie to Databricks Workflows. Lakeflow Jobs, previously called Databricks Workflows, runs a DAG of tasks with dependencies, retries, if/else and for-each control flow. Shell, Hive, Spark and Java actions map to notebook, Python script, SQL, JAR and Spark Submit tasks. Coordinators map to triggers: scheduled for time-based runs, file arrival where Oozie waited on a data-availability check, table update, or continuous.

Impala to Databricks SQL. Point interactive and BI traffic at a SQL warehouse and retest each query against the same data. Rank queries by how often they run and move the busiest first.

Hive and MapReduce. Hive SQL translates to Spark SQL with minimal refactoring, per Databricks. MapReduce jobs are rewritten as Spark. Where Kafka feeds the cluster, ingestion becomes Structured Streaming into Delta; the Kafka to Databricks post covers it.

Ranger on CDP and Sentry on CDH both say who can read which table, column and row. In Unity Catalog the same intent becomes grants to account-level groups:

GRANT USE CATALOG ON CATALOG main TO `analysts`;
GRANT USE SCHEMA ON SCHEMA main.sales TO `analysts`;
GRANT SELECT ON TABLE main.sales.orders TO `analysts`;

A user needs USE CATALOG and USE SCHEMA on the parents as well as the privilege on the object. Column masking and row-filter policies become Unity Catalog column masks and row filters.

Do not convert policies one by one. Reduce the exported policy set to the groups and tables that are actually used, then write grants for those. Kerberos keytabs go away with the cluster, and identities are managed at the Databricks account level.

Pick two or three pilot workloads with clear owners and run them on both platforms, comparing latency, throughput and data accuracy. Databricks recommends automating row-count, checksum and schema checks at every stage. Add aggregates by partition and by business key, because row counts alone miss type coercion and timestamp changes.

Give each wave a hard end date for the parallel run. Dual-platform spend is the cost that quietly grows.

Migrate in waves, high-value and low-complexity pipelines first. For each wave: freeze writes on the source dataset, run a final incremental distcp -update, convert or register the tables, point consumers at the Unity Catalog names and leave the old path read-only for a fixed period. If you used fallback mode on external locations during the move, turn it off at the end.

Retire the cluster only when no approved consumer remains. That condition, not the calendar, is the finish line.

• ORC and SerDe tables found late. CONVERT TO DELTA and SYNC both refuse them, so find them in the inventory.
• Shared directories. Several external tables over one path break the others on conversion and trip LOCATION_OVERLAP on sync.
• Writers still running. Appends during conversion create files Delta cannot see.
• Small files. They arrive with the data. Compact them with OPTIMIZE after conversion.
• Policies copied, not reduced. Years of Ranger or Sentry sprawl becomes years of grant sprawl.
• Unowned jobs. A scheduled job nobody owns fails silently after the cluster is gone.

If you want a second pair of hands on the inventory or the cutover plan, see our Databricks migration services. Clusters that must keep running while you move still need care, and Hadoop consulting and support covers NameNode, YARN and Kerberos until the last workload is off.

Sources: Databricks Hadoop migration guidance, CONVERT TO DELTA, SYNC, Hive metastore federation, UCX, Lakeflow Jobs, job triggers, Unity Catalog privileges and the 13 August 2026 press release; Apache Hadoop DistCp guide; Cloudera on Sentry to Ranger. Check the docs for your Databricks Runtime version before a migration window.