01 Foundations¶
The ideas every other section builds on: how analytical systems differ from operational ones, where analytical data is kept, how it is moved and processed, and how a platform is organised into layers and owned by teams.
Before starting
nothing. This is the first section.
Pages, in reading order¶
- OLTP vs OLAP: OLTP (online transaction processing) systems run the business one record at a time; OLAP (online analytical processing) systems answer questions across many records at once.
- Data lake, warehouse and lakehouse: Three ways to build an analytical store: a warehouse keeps data in a database that manages its own storage, a lake keeps files in cheap object storage, and a lakehouse adds table semantics to the lake so it can work like a warehouse.
- ETL, ELT and EtLT: Three orderings of the same steps, extract, transform and load, that differ in where data is reshaped: before it reaches the analytical store, inside it, or lightly before loading and fully inside.
- Processing patterns: batch, micro-batch and streaming: Data can be processed in large scheduled chunks, in small frequent chunks, or continuously as each event arrives; the choice trades freshness against cost and complexity.
- Schema-on-write and schema-on-read: Schema-on-write checks data against a fixed structure when it is stored; schema-on-read stores data as it comes and applies a structure only when it is queried.
- Medallion layers: bronze, silver and gold: A way to organise a data platform into three layers of increasing quality: bronze holds data as it arrived, silver holds it cleaned and conformed, and gold holds it shaped for specific uses.
- Data products and data mesh: A data product is a dataset treated like a product, with an owner, a promise to its consumers and a clear interface; data mesh is an approach that lets the teams who know the data own and publish such products themselves.
- Data team roles (optional): The jobs people do on a data platform, such as data engineer, analytics engineer, analyst, data scientist, ML engineer, platform engineer and data product owner, what each is responsible for, and where they overlap.