# 08 Data science and ML

How data becomes analysis, experiments and models: the data science workflow, working with lake data from Python, controlled experiments, features and training data, and taking models to production and keeping them accurate.

> [!NOTE]
>
> **Before starting**
>
> [04 Modelling and transformation](<https://wiki.4tm0.net/data/section-modeling.html>), [05 Governance and security](<https://wiki.4tm0.net/data/section-governance.html>) and [06 Serving](<https://wiki.4tm0.net/data/section-serving.html>), in particular [SQL query engines](<https://wiki.4tm0.net/data/sql-query-engines.html>), [Dimensional modelling](<https://wiki.4tm0.net/data/dimensional-modeling.html>), [History and slowly changing dimensions](<https://wiki.4tm0.net/data/history-and-scd.html>), [Semantic layer](<https://wiki.4tm0.net/data/semantic-layer.html>), [Data contracts](<https://wiki.4tm0.net/data/data-contracts.html>), [PII and data classification](<https://wiki.4tm0.net/data/pii-and-classification.html>) and [Low-latency serving stores](<https://wiki.4tm0.net/data/low-latency-serving-stores.html>) (optional in 06, but needed here).

## Pages, in reading order

1. [The data science workflow](<https://wiki.4tm0.net/data/data-science-workflow.html>): How a business question becomes a trustworthy answer: framing the question, exploring the data, testing explanations, and deciding whether the result is a one-off analysis, a recurring report or a model.
2. [Python on the lakehouse](<https://wiki.4tm0.net/data/python-on-the-lakehouse.html>): Working with lakehouse tables from Python: notebooks for exploration, dataframe libraries for analysis and feature work, and the choices that decide where the computation runs.
3. [Experimentation and A/B testing](<https://wiki.4tm0.net/data/experimentation-ab-testing.html>): Measuring the effect of a change by randomly splitting users into groups that see different versions, and comparing their outcomes.
4. [Feature engineering and feature stores](<https://wiki.4tm0.net/data/feature-engineering-feature-stores.html>): Feature engineering turns raw data into the inputs a model learns from; a feature store keeps those definitions in one place and provides the same values for training and for prediction.
5. [Training data management](<https://wiki.4tm0.net/data/training-data-management.html>): Building, versioning and governing the datasets models are trained and evaluated on: what goes in, how it is split, how labels are made, and whether the data may be used for training at all.
6. [The ML lifecycle and MLOps](<https://wiki.4tm0.net/data/ml-lifecycle-mlops.html>): The path from a training dataset to a model whose predictions people and systems use, and MLOps, the practices that make that path repeatable, traceable and safe to change.
7. [Model serving and monitoring](<https://wiki.4tm0.net/data/model-serving-monitoring.html>): Running a trained model to produce predictions, in batches or per request, releasing new versions safely, and watching whether its predictions stay accurate as the world changes.

## Previous and next section

[07 Operations](<https://wiki.4tm0.net/data/section-operations.html>) · [09 Generative AI](<https://wiki.4tm0.net/data/section-genai.html>)