Data lake, warehouse and lakehouse¶

Three ways to build an analytical store: a warehouse keeps data in a database that manages its own storage, a lake keeps files in cheap object storage, and a lakehouse adds table semantics to the lake so it can work like a warehouse.

Why it matters¶

The choice decides who can read the data, which engines can process it, what it costs to keep, and how much work it takes to keep it correct. Many new platforms are built as lakehouses, and the reasons make more sense once the two older approaches are clear.

How it works¶

A data warehouse is a database built for analysis. Data is loaded into it, stored in the warehouse's own internal format, and queried with SQL through that one product. The warehouse handles everything: storage layout, transactions, statistics, access control. Because one product controls both storage and execution, it can guarantee transactions and optimise queries end to end. Older warehouses also tied storage and compute to the same machines; cloud warehouses separate them internally and scale them independently. The cost is that the data is reachable only through that product, and unstructured data such as images, audio or PDFs has no natural home.

A data lake is a large store of files, usually object storage. Anything can be written there in any format: CSV exports, JSON events, Parquet files, images. Storage is cheap and any tool that reads the file format can process it. But a folder of files is not a table. There are no transactions, so a reader can see a half-written load; no schema enforcement, so files drift apart; and no simple way to update or delete a row. Lakes built this way often became hard to trust, and were nicknamed data swamps.

A lakehouse keeps the data in open files in object storage, like a lake, and adds an open table format and a catalog on top, so the files behave like tables. The table format, such as Apache Iceberg, Delta Lake or Apache Hudi, records which files make up a table at each moment and makes changes atomic. The catalog maps table names to their current state. Any engine that understands the format can read and write the same tables: a SQL engine for reports, a Python notebook for data science, a streaming job for ingestion. Storage and compute are fully separated: the data is stored once, and each engine brings its own compute, which can be scaled, paid for or replaced on its own.

flowchart LR
  subgraph WH[Warehouse]
    direction TB
    E1[Warehouse engine] --- S1[(Internal storage)]
  end
  subgraph LK[Lake]
    direction TB
    T1[Tool A] --> F1[(Files in object storage<br/>no table rules)]
    T2[Tool B] --> F1
  end
  subgraph LH[Lakehouse]
    direction TB
    E2[SQL engine] --> C[Catalog:<br/>table name to current state]
    E3[Notebook] --> C
    E4[Streaming job] --> C
    C --> TF[Table format metadata:<br/>which files, schema, snapshots]
    TF --> F2[(Open files in object storage)]
  end
  WH ~~~ LK ~~~ LH

Many warehouses can now read and write open table formats, and lakehouse engines offer warehouse features such as fine-grained access control. The useful question is less which label applies and more where the data physically lives, in what format, and which engines can reach it.

Key terms¶

• Data warehouse: an analytical database that manages its own storage format.
• Data lake: files of any format in object storage, without table rules.
• Lakehouse: open files in object storage plus an open table format and a catalog.
• Open table format: a specification that turns a set of files into a table with transactions and a schema.
• Separation of storage and compute: data stored once, processed by engines that can be scaled or swapped independently.

Example¶

The shop's orders, in each approach. The lake and the lakehouse can hold the same Parquet files; the difference is the metadata and catalog on top.

Pitfalls¶

• Treating a lake as a dumping ground. Without table formats, ownership and quality rules, files accumulate faster than anyone can explain them.
• Assuming open formats mean no lock-in. The catalog, access rules and engine-specific features can still tie you to a product.
• Choosing by label. Compare cost, engine choice, governance and the skills of the team, not the marketing category.
• Running several engines without one catalog. Each engine then sees its own version of the truth.

Learn more¶

• Armbrust et al., Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics, CIDR 2021: https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf
• Apache Iceberg, table spec overview: https://iceberg.apache.org/spec/

Related¶

ETL, ELT and EtLT · Object storage and the S3 API · Apache Iceberg: table format basics · SQL query engines