01
Hadoop and Cloudera Estates
HDFS data, the Hive metastore, Impala and Spark workloads, Oozie schedules and Ranger or Sentry policies.
Databricks Migration
Databricks Migration Services for Hadoop, Cloudera, Snowflake and BigQuery
Databricks migration services move your data, jobs, SQL and permissions from Hadoop, Cloudera, Snowflake or BigQuery onto Databricks, with every workload run in parallel against the old platform until the numbers match. AceMQ starts with a written assessment, so you see the scope, the order of work and a quote before you commit to a migration.
- source platforms: Hadoop, Cloudera, Snowflake, BigQuery
- 4
- phases, each with dual-run validation before cutover
- 5
- hours to a written quote
- 24
Is This You?
You probably need this if…
- 01
Your Hadoop or Cloudera cluster faces a hardware refresh or licence renewal nobody wants to sign
- 02
Snowflake or BigQuery handles the warehouse, but data engineering and machine learning run somewhere else
- 03
Hive tables, Oozie workflows and Spark jobs have piled up for years and there is no current inventory
- 04
The real access rules live in Ranger or Sentry policies and nowhere else
- 05
Reporting cannot go dark, so a big-bang cutover is off the table
- 06
You want the scope, order of work and a quote in writing before budget is committed
Outcomes
Where you are now, and where you end up
Concrete state changes, not deliverable counts. This is what actually differs about your Databricks estate when the engagement closes.
Before
Nobody can list every table, job, UDF and consumer on the old platform.
After
A written inventory of data, pipelines, SQL and users, with each workload given a migration path and a place in the order.
Before
The target is a vague "lakehouse" and workloads move before storage, catalogs and identity are designed.
After
A target design agreed first: cloud storage layout, Unity Catalog catalogs and schemas, compute types and identity.
Before
Cutover rests on someone saying the new numbers look right.
After
Dual-run validation per workload: row counts, aggregates and query results compared against the source before anything is switched.
Before
The old platform stays on indefinitely because nobody dares to switch it off.
After
A cutover plan with a rollback window and a decommission step for each wave of workloads.
Scope
What's covered
- 02
Snowflake and BigQuery Estates
Tables, views, SQL, scheduled queries and BI connections.
- 03
Data Movement
HDFS to cloud object storage, Parquet and ORC to Delta Lake, and ongoing ingestion rebuilt with Auto Loader or Structured Streaming.
- 04
Hive Metastore to Unity Catalog
Table upgrade, catalog and schema design, and a plan for tables that must stay queryable meanwhile.
- 05
SQL and Code Conversion
HiveQL, Impala, Snowflake and BigQuery SQL, Hive UDFs and Spark code recompiled against Databricks Runtime. Every converted script is tested.
- 06
Orchestration
Oozie coordinators rebuilt as Lakeflow Jobs, or your existing orchestrator kept.
- 07
Security and Access
Ranger and Sentry policies and Kerberos groups re-expressed as Unity Catalog grants to groups from your identity provider.
- 08
Validation and Cutover
Dual runs, reconciliation reports, a rollback window and source decommissioning once each wave is signed off.
The Engagement
How it actually runs
Every phase has a defined duration and a concrete artifact handed over at the end of it. You always know what stage you're in and what you've received.
- Phase 1Scoped per estate
Assessment
The entry point. We inventory what exists, rate each workload by effort and set the order of work, so you have scope, a wave plan and a written quote before committing budget.
You receive
- Inventory of tables, jobs, UDFs, users and consumers
- Workloads rated by complexity, with a path for each
- Wave plan and written quote
- Phase 2Agreed at assessment close
Target Design
The decisions that are expensive to reverse, made first: storage layout, Unity Catalog structure, compute policies, networking and identity.
You receive
- Target architecture
- Catalog, schema and naming model
- Compute and access design
- Phase 3One representative workload
Pilot
One real workload migrated end to end, with its dual run and validation report. It tests the design and corrects the estimate before the waves begin.
You receive
- Migrated pilot workload
- Validation report
- Revised wave plan
- Phase 4Per wave, set in the plan
Migration Waves
In each wave the new job runs beside the old one, and its outputs are reconciled against the source until a data owner on your side signs off.
You receive
- Converted jobs and loaded data per wave
- Reconciliation report per workload
- Sign-off record
- Phase 5Per wave, with a rollback window
Cutover and Decommission
Consumers are switched, the source is kept through the rollback window, then retired.
You receive
- Cutover and rollback plan
- Runbooks
- Decommission checklist for the source platform
Hadoop to Databricks migration: what each component becomes
A Hadoop to Databricks migration replaces the cluster's storage, metastore, engines and schedulers with cloud object storage, Unity Catalog, Databricks compute and Lakeflow Jobs. This is where each component usually lands.
| Source component | Databricks target | What changes |
|---|---|---|
| HDFS | Cloud object storage (Amazon S3, Azure Data Lake Storage Gen2 or Google Cloud Storage) | Data is copied out of HDFS. Parquet and ORC are converted to Delta Lake, and partition layouts built for HDFS are usually redesigned. |
| Hive metastore | Unity Catalog (catalog.schema.table) | Tables are upgraded or registered. Hive metastore federation can keep not-yet-migrated tables queryable during the move. |
| HiveQL | Spark SQL on Databricks Runtime | Most HiveQL runs with limited refactoring. SerDes, Hive UDFs and some DDL statements need changes. |
| Impala | Databricks SQL warehouses | Queries are re-pointed and tuned, and BI connections move to the warehouse. |
| Spark on YARN | Databricks compute | Spark code is recompiled against Databricks Runtime libraries. YARN queues become compute policies and separate compute. |
| Oozie | Lakeflow Jobs | Workflows and coordinators are rebuilt as jobs with tasks and triggers. This is a rewrite, not a conversion. |
| Ranger or Sentry policies | Unity Catalog grants, row filters and column masks | Policies are re-expressed as grants to account-level groups. |
| Kerberos and LDAP groups | Your identity provider, synced to the Databricks account | Groups are provisioned with automatic identity management or SCIM. |
In Hadoop and Cloudera migrations, the work that takes longest is rarely the data copy. Databricks' own migration guidance names the usual suspects: Hive SerDes that must move to Databricks-native codecs, Hive UDFs that must be installed as libraries or refactored to native Spark, and table directory structures that Databricks uses differently. The same guidance calls over-partitioned tables a main cause of slow workloads on Delta Lake, so we redesign layout instead of carrying it over. We size a migration on those items, not on terabytes.
Still running Hadoop and need it supported during the move? See Hadoop consulting and support
For a service-level example of this work, see the Hadoop to lakehouse migration use case
Snowflake to Databricks and BigQuery to Databricks migration
Snowflake and BigQuery migrations move tables, views and SQL onto Delta tables and Databricks SQL warehouses. The work is larger than it looks in SQL and consumer terms, because every dashboard and downstream job has to be re-pointed.
| Source workload | Databricks target | Phased-cutover approach |
|---|---|---|
| Snowflake tables and views | Delta tables in Unity Catalog, queried through Databricks SQL warehouses | Lakehouse Federation exposes Snowflake as a read-only foreign catalog, so consumers can move in waves while data is copied. |
| BigQuery datasets | Delta tables in Unity Catalog, queried through Databricks SQL warehouses | Query federation to BigQuery works the same way, read-only, through a connection and a foreign catalog. |
| Snowflake and BigQuery SQL | Databricks SQL (Spark SQL, mostly ANSI-compliant) | Dialect functions are converted and tested. Transactions are table-level on Databricks, and primary and foreign key constraints are informational. |
| Scheduled queries and tasks | Lakeflow Jobs | Rebuilt as jobs with tasks and triggers, then dual-run against the source schedule. |
| BI and downstream tools | Databricks SQL warehouses | Connections are re-pointed one dashboard group at a time, with query performance retested. |
Databricks also offers a Beta agentic code converter for SQL from dialects such as T-SQL, Snowflake, Oracle, BigQuery and Teradata. We treat its output as a draft and validate it like hand-converted code.
Databricks' documentation recommends its ingestion connectors over federation when volume and latency matter, so we use federation to bridge the cutover, not as the end state. We also say when a move is not worth making: if SQL warehousing is all you run and it works, the assessment will tell you.
Your Snowflake estate stays on Snowflake? See Snowflake consulting and support
Still choosing between the platforms? Read Databricks vs Snowflake compared
How dual-run validation works
In a dual run the old and the new pipeline process the same inputs side by side, and their outputs are compared before any consumer is switched.
- Data checks — Row counts, key-level comparisons and aggregates, table by table.
- Query checks — Real production queries run on both platforms and the results compared.
- Job checks — Run time, failures and cost of the new job against the old one.
- Access checks — Real user groups tested against the new grants, including what they must not see.
Rule we hold
No workload is cut over until a data owner on your side has signed its validation report, and the source stays available through the rollback window.
The practice is led by a former Databricks professional-services resident solution architect, and AceMQ's delivery team manages scope, schedule and reporting.
Where this migration work stops
- Independent of Databricks — AceMQ does not resell Databricks or cloud consumption. Databricks and your cloud provider bill you directly, and nothing on this page is Databricks guidance or endorsement.
- Reports and dashboards — We migrate pipelines, SQL and permissions. Rebuilding reports in your BI tool is your team's job or scoped separately.
- HBase, Kudu and other operational stores — These have no one-to-one Databricks product. The assessment flags them and recommends a target, and the decision stays with you.
- Sign-off stays with you — Your data owners know what a correct number looks like, and they approve each wave.
After cutover, keep spend under control with Databricks cost optimization
Building the landing zone from scratch first? See Databricks implementation and Unity Catalog setup
For production cover once you are live, see Databricks support
Customer Success
Real Databricks Results
See how enterprises trust AceMQ for their most critical Databricks workloads.
Databricks DBU Cost Governance
Global Payments Processor
Right-sizing Databricks compute by moving scheduled work off all-purpose clusters and tightening autoscaling, instance selection, and idle timeouts.
Databricks
Apache SparkDelta Lake
TerraformDatabricks Unity Catalog Migration
Healthcare Analytics Provider
Migrating off the legacy Hive metastore to Unity Catalog with external location mapping, table upgrades, and a grant model that survives audit.
DatabricksDelta LakeApache SparkTerraformDatabricks Delta Small-File Remediation
Digital Retail Platform
Fixing Delta tables where streaming writes and over-partitioning have produced millions of tiny files, stalling reads and vacuum operations.
DatabricksDelta LakeApache SparkDatabricks Job Cluster Failure Support
Insurance Services Group
Named-engineer support for production Databricks job failures — driver OOM, spot reclamation, library conflicts, and workflow retry storms.
DatabricksApache SparkDelta LakeApache Spark Workload and Cost Assessment
Media Streaming Provider
Profiling a Spark estate to find over-provisioned jobs, redundant pipelines, and workloads better served by something other than Spark.
Apache SparkDatabricksDelta Lake
Kubernetes
FAQ
Questions about Databricks migration
What do Databricks migration services include?
An assessment of the source platform, a target design on Databricks, a pilot, migration in waves with dual-run validation, and cutover and decommissioning. Data, jobs, SQL and permissions are in scope. Reports in your BI tool are scoped separately.
How do you migrate from Hadoop or Cloudera to Databricks?
We inventory the cluster, design the target in cloud storage and Unity Catalog, and move workloads in waves. Data goes to cloud object storage as Delta Lake, Hive tables are upgraded to Unity Catalog, code is adapted and Oozie schedules are rebuilt as Lakeflow Jobs. Each wave is dual-run against the cluster before cutover.
Can Snowflake or BigQuery be migrated without a big-bang cutover?
Yes. Lakehouse Federation lets Databricks query Snowflake or BigQuery in place, read-only, through Unity Catalog, so consumers can move in waves while data is copied and validated. The final cutover is then small and reversible.
How long does a Databricks migration take?
It depends on the number of workloads and how much of the estate is Hive UDFs, custom schedulers and undocumented policy. We do not quote a duration before seeing the estate. The assessment produces a wave plan with a timeline per wave.
Do we have to move from the Hive metastore to Unity Catalog?
Databricks includes a Hive metastore by default, but Unity Catalog is the governance layer we design for. Tables can be upgraded with UCX, the upgrade wizard, SYNC or CREATE TABLE CLONE, or mirrored with Hive metastore federation during the transition. UCX is a Databricks Labs project that Databricks does not support directly, so we review its output before acting on it.
What does a Databricks migration cost?
Scope drives cost: the number of workloads, the source platform and the validation required. We do not publish a price list. Send the source platform, a rough count of tables and jobs and your target cloud, and we will send a written quote within 24 hours.
Let's talk
Start with a Databricks Migration Assessment
Tell us the source platform, a rough count of tables and jobs, and your target cloud. We will scope the assessment and send a written quote within 24 hours.
Get in Touch
Talk to a Databricks Expert
Send us a message and we'll follow up within one business day — or book a free 30-min consultation directly.
305-204-2607info@acemq.com66 W. Flagler St. 9th Floor
Miami, FL 33130
Miami, FL 33130
Prefer to talk now? Call us directly or use the consultation tab to find a time that works.