02 Storage¶
How a lakehouse physically stores data: object storage, columnar files and partitioning at the bottom, the Iceberg table format on top of them, the jobs that keep tables healthy, and the catalog through which every engine finds a table.
Before starting
01 Foundations, in particular Data lake, warehouse and lakehouse. Query performance on lake tables also uses terms from SQL query engines.
Pages, in reading order¶
- Object storage and the S3 API: Object storage keeps data as whole objects, each a sequence of bytes plus a little metadata, addressed by a bucket and a key and read or written over HTTP.
- Columnar files and Parquet: Parquet is an open file format that stores a table's values column by column, encoded and compressed, with statistics that let readers skip the parts they do not need.
- Partitioning: Partitioning splits a table's files into groups by the value of a key, such as the day, so that a query filtering on that key reads only the matching groups.
- Apache Iceberg: table format basics: Iceberg is an open table format: a set of rules for storing a table as plain files in object storage so that many engines can read and write it safely, as if it were a database table.
- Iceberg maintenance: Regular jobs that keep an Iceberg table fast and its storage under control: compacting small files, expiring old snapshots and removing files that nothing references.
- Iceberg internals: manifests and planning (optional): How Iceberg's metadata files are structured, and how an engine uses them to find the files a query needs without listing storage.
- Query performance on lake tables: Query performance on lake tables comes mostly from reading less: fewer files, fewer row groups and fewer columns.
- Catalogs: A technical catalog is the service that maps table names to each table's current metadata, coordinates commits, and controls who may read or write each table.
- Table formats compared: Iceberg, Delta Lake, Hudi and Paimon (optional): The main open table formats solve the same problem, turning files into tables, with different designs and ecosystems; this page compares them and explains how they interoperate.