Training data management¶

Building, versioning and governing the datasets models are trained and evaluated on: what goes in, how it is split, how labels are made, and whether the data may be used for training at all.

Why it matters¶

A model is a function of its training data. If nobody can say exactly which data a model was trained on, its behaviour cannot be explained, reproduced, audited or fixed. Many legal and ethical questions about AI are about training data: whose it is, whether they agreed, and what it over- or under-represents.

How it works¶

A training dataset has three parts: examples, the entities and times to predict for; features, the inputs for each example; and labels, the outcomes the model should learn, such as whether the customer churned. It is built by a tested, owned pipeline like any other table, not assembled once in a notebook.

Versioning makes it reproducible. In a lakehouse, a training set is a table, and a table version is a snapshot. A training run records the snapshot IDs of the training and evaluation tables, but snapshots expire with normal retention, so the pipeline also creates a tag on the table, such as forecast-train-2026-10-26, retained as long as the model is in use. Time travel to that tag rebuilds exactly the data the model saw. Large file collections, such as images, are versioned by a manifest table of files and checksums.

Splitting decides how the model is evaluated. Data is divided into training, validation (for choosing settings) and test sets (for the final, untouched evaluation). For data with time, splits should follow time: train on earlier periods, test on later ones, as the model will be used. Random splits on time-ordered data let information from the future leak into training. When the model must predict for new entities, such as new customers, keep each entity in one split so it cannot memorise them; when it forecasts the same entities over time, as below, the time split is what matters.

Labels need care. Some come from data, such as whether an order was returned; they must be defined precisely and only known after the prediction time. Others need people, such as classifying support tickets or marking product images; labelling guidelines, agreement checks between labellers, and a record of who labelled what make them trustworthy. Class balance matters: if 2% of customers churn, a model can be 98% accurate by predicting nobody churns, so evaluation uses measures suited to imbalance.

Synthetic data, generated to resemble real data, can fill gaps or protect privacy in development, but inherits the biases of its generator and does not replace evaluation on real data.

Rights decide what may enter at all. Personal data used for training needs a lawful purpose, which may differ from its collection purpose. Classification tags and consent fields decide which rows and columns may enter a training set, and erasure requests must reach training sets and, through policy, the models trained on them. A tag keeps its files past snapshot expiry, so a tagged training set with personal data must be rewritten and re-tagged when an erasure arrives, or built without identifiers. Licences matter for external data: public availability does not imply permission to train.

flowchart LR
  F[(Feature tables)] --> B[Build training set:<br/>examples, features,<br/>labels]
  L[(Labels:<br/>from data or people)] --> B
  P{Usage rights:<br/>consent, classification,<br/>licence} --> B
  B --> T[(training_set table)]
  T -->|tag: forecast-train-2026-10-26| V[Versioned for<br/>reproducibility]
  T --> S[Time-based split:<br/>train, validation, test]

Key terms¶

• Example: one case to predict for, with an entity and a time.
• Label: the known outcome a model learns to predict.
• Training, validation and test sets: data for learning, for tuning, and for final evaluation.
• Time-based split: splitting by date so the model is tested on later data than it learned from.
• Class imbalance: when one outcome is much rarer than others.
• Synthetic data: generated data that imitates real data.
• Dataset version: an identifiable, reproducible state of a dataset, such as a table tag.

Example¶

The training set for the shop's demand forecast:

• Examples: each product and day from January 2024 to September 2026; the label is units sold over the following 14 days.
• Features: from feat_product_daily, joined point in time.
• Split: train on 2024 to March 2026, validate on April to June 2026, test on July to September 2026, with a 14-day gap between periods, the length of the label window.
• Rights: the data is aggregated sales, no personal data, so no consent filter is needed.
• Version: the pipeline writes shop.ml_demand_training and tags the snapshot forecast-train-2026-10-26; the registry entry for the new model stores the tag.

A month later, one category shows strange forecasts. Time travel to the tag shows its training rows carried a wrong category from dim_product, so the fix and retraining trace to the exact data.

Pitfalls¶

• Training sets built by hand in notebooks. Nobody can rebuild or audit them.
• Random splits on time series. The test score is too optimistic.
• Labels that use future information. They are not available when the model predicts.
• Accuracy on imbalanced data. Use measures that reflect the rare class.
• Assuming data may be used for training because it is stored. Check purpose, consent and licence.

Learn more¶

• Gebru et al., Datasheets for Datasets: https://arxiv.org/abs/1803.09010
• Apache Iceberg, branching and tagging: https://iceberg.apache.org/docs/latest/branching/

Related¶

The ML lifecycle and MLOps · Write-audit-publish · Responsible AI and AI governance · Fine-tuning or RAG