Lakehouse architecture in short

A lakehouse architecture keeps data as open files on object storage, adds a table format that gives those files database-style guarantees, and lets several compute engines read the same tables. It has four parts: storage, a table format, compute, and the pipelines and governance that keep the tables correct. The first three are products you choose. The fourth is work you do.

This article explains each part and where teams get into trouble. If you are choosing a stack or retiring an older platform, our data engineering consulting page describes the scope of that work.

A data lake is files on object storage. It holds structured, semi-structured and unstructured data cheaply and flexibly, but files alone give you no transactions, so concurrent writers and failed jobs can leave half-written results. A data warehouse stores structured data in its own managed format and gives strong SQL guarantees, at the price of loading data into it and, usually, one engine. A data lakehouse keeps the lake's open files and adds the table format, so structured tables get warehouse-style guarantees while the files stay readable by other engines. Unstructured data such as images and logs can sit alongside the tables, but the guarantees apply to the tables.

A table format is what turns a folder of files into a table. Delta Lake is an open source storage framework for building a lakehouse on data lakes such as S3, ADLS, GCS and HDFS. Its documentation lists ACID transactions with serializable isolation on Spark, scalable metadata handling, time travel, schema enforcement, unified batch and streaming, and merge, update and delete operations. Apache Iceberg is a table format for large analytic datasets, and its documentation lists support from Spark, Flink, Kafka Connect and Hive, and from third-party engines including Trino, Presto, Snowflake, BigQuery, Athena, EMR, Redshift and DuckDB.

The practical question is which engines must read the table. If one engine writes and everything else reads through it, either format works. If several engines must write, check the connector support for each, version by version. Do not run both formats without a reason, because each needs its own maintenance and governance.

Most lakehouses organise tables in layers. Databricks calls this the medallion architecture and describes it as a series of data layers that denote the quality of the data in the lakehouse. Bronze keeps raw, unvalidated data in its original form so it can be reprocessed and audited. Silver holds validated, cleaned and enriched data, with deduplication, normalisation and schema enforcement. Gold holds aggregated, business-aligned data for reporting, dashboards and machine learning.

The layers are a convention, not a product. The value is that each hop has one job and a check at its boundary. The cost is that every hop is a pipeline someone must run. Databases feed bronze most cleanly through change data capture, which delivers deletes and ordering that a nightly extract loses.

• Table maintenance. File compaction, removal of unreferenced files and statistics still need a schedule or an automatic service, and the services differ by platform. A table that is never maintained slows down quietly.
• Cost control. Open storage is cheap. Compute is not, and idle or oversized clusters are the usual bill. A Spark health check looks at this for Spark estates.
• Governance. Open files can be read by anything with storage access. Access rules belong in a catalog, and bucket permissions must not bypass it.
• Reliable pipelines and ingestion. Data ingestion, idempotent tasks, safe backfills and alerts with an owner are the same problems as in any pipeline estate.

A migration is easiest when the layers are decided first and the old and new paths run side by side until the numbers match. The Hadoop to Databricks migration guide walks through the staging, and Databricks vs Snowflake compares two common targets. For a Databricks build, see Databricks implementation.

• Delta Lake documentation
• Apache Iceberg documentation
• Databricks documentation: What is the medallion lakehouse architecture?

Documentation read on 31 October 2026. Engine and connector support changes by release, so check the pages for your versions.