# Glossary Every key term from the topic pages, one line each, with a link to the page that explains it. Generated by build\_reference.py. ## A - ABAC: permissions decided by rules over user, resource and context attributes. See [RBAC and ABAC](). - Access review: a periodic check of who has access against who needs it. See [Audit logging](). - Accuracy: agreement with reality or a trusted source. See [Data quality dimensions](). - Acting as the user: calling downstream systems with the user's identity, not a service's. See [Agents and MCP](). - Activation: making data usable where actions happen. See [Reverse ETL and activation](). - Activity schema: a model with one table of entity activities over time. See [Other modelling approaches: Data Vault, one big table and activity schema](). - Adapter: the plugin that connects dbt to an engine. See [dbt](). - Additive, semi-additive, non-additive: how a measure can be summed. See [Dimensional modelling](). - Aggregate: the business object an event is about, such as one order. See [The outbox pattern](). - AI agent: a model-driven application that calls tools to answer or act. See [Agents and MCP](). - AI inventory: a register of all AI systems with owners, purposes and risk levels. See [Responsible AI and AI governance](). - Allowed lateness: how long a window still accepts updates after its watermark has passed. See [Late and out-of-order data](). - Analytics engineer: builds tested models and metrics for consumers. See [Data team roles](). - Anomaly detection: flagging values that differ from their usual pattern. See [Data quality dimensions](). - Anonymisation: making identification no longer possible. See [PII and data classification](). - Apache Arrow: a columnar in-memory format shared by many Python data tools. See [Python on the lakehouse](). - Apache XTable: a project that translates metadata between table formats. See [Table formats compared: Iceberg, Delta Lake, Hudi and Paimon](). - API contract: the stable definition of endpoints and fields that clients rely on. See [Data APIs and the backend-for-frontend pattern](). - Append mode: every record becomes a new row. See [Writing to Iceberg from Kafka](). - Approximate nearest neighbour (ANN): fast search that finds close vectors with high, not perfect, accuracy. See [Embeddings, vector search and RAG](). - Asset: a data object, such as a table, that the orchestrator tracks and keeps up to date. See [Orchestration](). - At-least-once: retried until acknowledged; may duplicate messages. See [Delivery guarantees and idempotency](). - At-most-once: never retried; may lose messages. See [Delivery guarantees and idempotency](). - Atomic commit: a pointer change that succeeds only if the table has not changed since the writer started. See [Catalogs](). - Atomic swap: replacing a published table with a staged one in one operation. See [Write-audit-publish](). - Audit: the checks run between writing and publishing. See [Write-audit-publish](). - Audit log: a record of security-relevant actions: who, what, which resource, when, result. See [Audit logging](). - Authentication: verifying who is making a request. See [Authentication and authorization: OIDC and JWT](). - Authorization: deciding whether a request is allowed. See [Authentication and authorization: OIDC and JWT](). ## B - Backend-for-frontend (BFF): an API dedicated to one client application. See [Data APIs and the backend-for-frontend pattern](). - Backfill: running a pipeline for past periods. See [Backfill and replay](). - Backward compatibility: new readers can read old data. See [Schema evolution](). - Batch serving: scoring on a schedule and writing predictions to a table. See [Model serving and monitoring](). - Bin-packing: compaction that combines files without reordering rows. See [Iceberg maintenance](). - Blocking test: a test whose failure stops publishing. See [Testing data pipelines](). - Bounded data: a finite set with a known end, as processed by a batch. See [Processing patterns: batch, micro-batch and streaming](). - Branch: a named, independent line of snapshots in a table. See [Write-audit-publish](). - Breaking change: a change that existing readers or writers cannot handle under the agreed compatibility mode, or that changes meaning; removing a field consumers use is breaking even when the registry allows it. See [Schema evolution](). - Breaking change (Data contracts): a change that can break consumers, requiring a new major version. See [Data contracts](). - Broker: the server that receives MQTT messages and forwards them to subscribers. See [IoT and device data](). - Bronze (raw): data as delivered, append-only, kept for rebuilding. See [Medallion layers: bronze, silver and gold](). - Bucket: a named container for objects, with a region and access policy. See [Object storage and the S3 API](). - Bucket versioning: keeping previous versions of overwritten or deleted objects. See [Disaster recovery](). - Bucketing: assigning rows to a fixed number of groups by a hash of a key. See [Partitioning](). - Business catalog: a searchable inventory of datasets with business context. See [Catalog and lineage](). - Business glossary: agreed definitions of business terms. See [Catalog and lineage](). - Business intelligence (BI): tools and practices for reporting and exploring business data. See [BI and dashboards](). - Business key: the identifier of an entity in the source, such as order\_id. See [Deduplication and merge](). - Business metadata: meaning and responsibility, such as descriptions, owners and terms. See [Catalog and lineage](). ## C - Canary release: sending a small share of traffic to a new version first. See [Model serving and monitoring](). - Cardinality: the number of distinct values of a column. See [Partitioning](). - Casting: converting a value to a declared type when reading or transforming. See [Schema-on-write and schema-on-read](). - Catalog: the service that knows where each table's current metadata.json is. Every commit goes through it. See [Apache Iceberg: table format basics](). - Catastrophic forgetting: a fine-tuned model losing general abilities it had before. See [Fine-tuning or RAG](). - CDC metadata columns: operation, source position and commit time kept alongside each change row. See [Writing to Iceberg from Kafka](). - Certification: marking a dashboard or dataset as trusted and supported. See [BI and dashboards](). - Change event: one row change, with before, after, operation and source metadata. See [Change data capture](). - Change-based sync: sending only rows that changed since the last sync. See [Reverse ETL and activation](). - Checkpoint: a consistent snapshot of state and source offsets, used to recover. See [Stream processing](). - Chunk: a section of a document, embedded and retrieved as a unit. See [Embeddings, vector search and RAG](). - Class imbalance: when one outcome is much rarer than others. See [Training data management](). - Classification level: a label, such as confidential, that decides handling rules. See [PII and data classification](). - Classification tag: a label on a column or table marking its sensitivity. See [Governance as code](). - Client credentials flow: how a service obtains a token as itself. See [Authentication and authorization: OIDC and JWT](). - CloudEvents: a CNCF specification for common event metadata. See [Event design](). - Clustering: placing similar values together, by sorting or Z-ordering, so statistics can skip. See [Query performance on lake tables](). - Column bounds: the lowest and highest value of a column in one data file. See [Iceberg internals: manifests and planning](). - Column chunk: the values of one column within one row group. See [Columnar files and Parquet](). - Column mask: an expression that hides or reduces a column's values for a user. See [Row-level security and masking](). - Column-level lineage: lineage between individual columns. See [Catalog and lineage](). - Column-oriented storage: all values of a column stored together. See [OLTP vs OLAP](). - Command: a request for something to happen, which may be refused. See [Event design](). - Commit interval: how often written files are committed to the table. See [Writing to Iceberg from Kafka](). - Compacted topic: a topic that keeps the latest record per key. See [Kafka and Kafka Connect](). - Compaction: rewriting small files, and their delete files, into fewer large files. See [Iceberg maintenance](). - Compatibility mode: the rule a registry enforces for a subject, usually one per topic key or value. See [Schema evolution](). - Completeness: no missing records or required values. See [Data quality dimensions](). - Concept drift: change in the relationship between inputs and outcomes. See [Model serving and monitoring](). - Concurrency: the number of queries served at the same time. See [Low-latency serving stores](). - Conditional write: a write that succeeds only if the object is absent or unchanged. See [Object storage and the S3 API](). - Conformed: using the same keys, names and codes across sources. See [Medallion layers: bronze, silver and gold](). - Conformed dimension: a dimension shared, with the same meaning, by several facts. See [Dimensional modelling](). - Confounder: a factor that influences both a cause and an effect, creating a false association. See [The data science workflow](). - Connector: a configured source or sink in Kafka Connect. See [Kafka and Kafka Connect](). - Connector (SQL query engines): a plugin that lets the engine read and write one kind of system. See [SQL query engines](). - Consistency: agreement of the same fact across tables and systems. See [Data quality dimensions](). - Consumer: uses data within agreed purposes and terms. See [Governance roles](). - Consumer (Data contracts): anyone who relies on the dataset. See [Data contracts](). - Consumer group: consumers that share the work of reading a topic, with their own committed offsets. See [Kafka and Kafka Connect](). - Continuous integration (CI): automatic checks run on every proposed change. See [Environments and CI](). - Control and treatment: the groups that see the current and the changed version. See [Experimentation and A/B testing](). - Control topic: the topic on which sink tasks and the coordinator exchange commit information. See [Writing to Iceberg from Kafka](). - Coordinator: the process that plans a query and schedules work. See [SQL query engines](). - Correlation ID: an ID that links events belonging to one process. See [Event design](). - Cosine similarity: a measure of how close two vectors point in the same direction. See [Embeddings, vector search and RAG](). - Cost attribution: assigning spend to teams, products or workloads. See [Cost and FinOps](). - Credential vending: the catalog issuing short-lived storage credentials scoped to one table. See [Catalogs](). - Cross-border transfer: data, or access to it, moving from one jurisdiction to another. See [Residency and regions](). - Cross-reference table: the record of which source IDs belong to which platform ID. See [Multi-source conformance](). - Crypto-shredding: making data unreadable by destroying its encryption key. See [Privacy lifecycle](). - Curated dashboard: a reviewed, owned dashboard intended as an official source. See [BI and dashboards](). - Cursor: a position token returned by an API to continue reading. See [Ingestion patterns](). - Cursor pagination: paging with a token that marks where the next page starts. See [Data APIs and the backend-for-frontend pattern](). - Customer-managed key: an encryption key controlled by the organisation, not the provider. See [Encryption and key management](). - Cutover: moving consumers from the old platform to the new one. See [Platform migration](). ## D - DAG: a directed acyclic graph of tasks and their dependencies. See [Orchestration](). - Data analyst: answers business questions with reports and analysis. See [Data team roles](). - Data API: an HTTP interface returning records from platform data. See [Data APIs and the backend-for-frontend pattern](). - Data contract: a machine-readable agreement on a dataset's schema, meaning, quality and service levels. See [Data contracts](). - Data custodian: operates the systems that store and move the data. See [Governance roles](). - Data diff: comparing the output of a changed model with production. See [Environments and CI](). - Data drift: change in the distribution of a model's inputs. See [Model serving and monitoring](). - Data engineer: builds ingestion and the reliable lower layers. See [Data team roles](). - Data lake: files of any format in object storage, without table rules. See [Data lake, warehouse and lakehouse](). - Data localisation: a legal requirement that certain data stays within a country. See [Residency and regions](). - Data minimisation: collecting and keeping only what is needed. See [PII and data classification](). - Data owner: accountable for a dataset; makes decisions about its use and quality. See [Governance roles](). - Data product: a published dataset with an owner, interface, contract and documentation. See [Data products and data mesh](). - Data product owner: accountable for a data product's value to its consumers. See [Data team roles](). - Data residency: the location where data is stored and processed. See [Residency and regions](). - Data scientist: uses statistics and models to investigate and predict. See [Data team roles](). - Data sovereignty: data being subject to the laws of where it is held, or of who holds it. See [Residency and regions](). - Data steward: manages definitions, access reviews and quality issues day to day. See [Governance roles](). - Data test: a query that returns rows breaking a business rule. See [Testing data pipelines](). - Data Vault: a modelling method using hubs, links and satellites for integration and history. See [Other modelling approaches: Data Vault, one big table and activity schema](). - Data warehouse: an analytical database that manages its own storage format. See [Data lake, warehouse and lakehouse](). - Dataframe: an in-memory table manipulated with code. See [Python on the lakehouse](). - Dataset version: an identifiable, reproducible state of a dataset, such as a table tag. See [Training data management](). - Datasheet for datasets: documentation of a dataset's origin, content and permitted uses. See [Responsible AI and AI governance](). - Declarative: describing the desired end state rather than the steps to reach it. See [Infrastructure as code](). - Declarative configuration: describing the desired state, and letting tools make the systems match it. See [Governance as code](). - Decommissioning: switching off the old platform once nothing uses it. See [Platform migration](). - Delete file: records which rows a later commit removed. Position deletes point at row positions; equality deletes match on column values. See [Apache Iceberg: table format basics](). - Deletion vector: a bitmap marking deleted rows of one data file, in format version 3. See [Iceberg maintenance](). - Deletion vector (Table formats compared: Iceberg, Delta Lake, Hudi and Paimon): a per-file record of deleted rows, used by Delta Lake and Iceberg v3. See [Table formats compared: Iceberg, Delta Lake, Hudi and Paimon](). - Denormalised model: related data combined into wider tables to avoid joins. See [OLTP vs OLAP](). - Deterministic matching: matching on exact, normalised identifiers. See [Multi-source conformance](). - Dialect translation: converting SQL and functions from one engine to another. See [Platform migration](). - Dimension: an attribute that metrics are grouped or filtered by. See [Semantic layer](). - Dimension table: descriptive context for facts, one row per member. See [Dimensional modelling](). - Direct identifier: a value that identifies a person on its own. See [PII and data classification](). - Distillation: training a smaller model to reproduce a larger model's outputs on a task. See [Fine-tuning or RAG](). - Domain: a business area with its own team, data and language, such as orders or payments. See [Data products and data mesh](). - Drift: a difference between declared and actual configuration. See [Governance as code](). - Drift (Infrastructure as code): differences between declared and actual infrastructure. See [Infrastructure as code](). - Dual write: writing to two systems separately, with no way to commit both together. See [The outbox pattern](). - Dynamic masking: masking applied at query time, depending on the user. See [Row-level security and masking](). ## E - Effective dates: valid\_from and valid\_to of a version. See [History and slowly changing dimensions](). - Egress: data transferred out of a region or cloud, usually charged. See [Cost and FinOps](). - Embedding: a vector representing the meaning of a piece of content. See [Embeddings, vector search and RAG](). - Encoding: a compact representation of values, such as dictionary or run-length. See [Columnar files and Parquet](). - Encryption at rest: protecting stored data. See [Encryption and key management](). - Encryption in transit: protecting data on the network, usually with TLS. See [Encryption and key management](). - Enforcement point: the component that applies filters and masks. See [Row-level security and masking](). - Entity: the thing features describe, such as a customer or product. See [Feature engineering and feature stores](). - Envelope: standard metadata fields carried by every event. See [Event design](). - Envelope encryption: encrypting data with data keys that are themselves encrypted by a master key. See [Encryption and key management](). - Environment: an isolated place to build and run pipelines, such as dev, CI or prod. See [Environments and CI](). - Erasure request: a request to delete all personal data about one person. See [Privacy lifecycle](). - Error budget: the amount of failure the SLO allows. See [Observability and SLOs](). - Evaluation set: versioned inputs with criteria or references for good outputs. See [LLM applications and evaluation](). - Event: a record of a fact that happened. See [Event design](). - Event ID: a unique, producer-generated ID used by consumers to deduplicate. See [The outbox pattern](). - Event listener: an engine plugin that receives events, such as completed queries. See [Audit logging](). - Event time: when the event happened. See [Late and out-of-order data](). - Exactly-once: each message affects the result once, through deduplication or transactions. See [Delivery guarantees and idempotency](). - Exactly-once processing: each event affects state once, even after failures. See [Stream processing](). - Example: one case to predict for, with an entity and a time. See [Training data management](). - Exchange: moving intermediate rows between workers, for example to group or join by key. See [SQL query engines](). - Expand and contract: making a breaking change in compatible steps. See [Schema evolution](). - Experiment tracking: recording parameters, data, metrics and outputs of every run. See [The ML lifecycle and MLOps](). - Explainability: the ability to give reasons for a model's predictions. See [Responsible AI and AI governance](). - Exploratory data analysis (EDA): examining data to understand it before testing hypotheses. See [The data science workflow](). - Export: a copy of data sent to someone, outside the provider's control. See [Data sharing](). - Exposure: the moment a unit actually experiences its variant. See [Experimentation and A/B testing](). - Extract: reading data out of a source system. See [ETL, ELT and EtLT](). - Extract (BI and dashboards): a copy of data stored in the BI tool and refreshed on a schedule. See [BI and dashboards](). ## F - Fact table: rows of events or measurements with numeric measures and dimension keys. See [Dimensional modelling](). - Failover: switching service to a replica in another location. See [Disaster recovery](). - Fairness check: comparing a model's performance or outcomes across groups. See [Responsible AI and AI governance](). - Fast-forward: moving a branch to a later snapshot of another branch. See [Write-audit-publish](). - Fault-tolerant execution: retrying failed parts of a query instead of the whole query. See [SQL query engines](). - Feature: an input variable to a model. See [Feature engineering and feature stores](). - Federated governance: shared rules agreed by domain and platform representatives, enforced automatically, with all other decisions left to domains. See [Data products and data mesh](). - Federation: one query reading from several systems through connectors. See [SQL query engines](). - File catalog table: a table with one row per file, describing and governing it. See [Unstructured and multimodal data](). - File skipping: dropping files whose column bounds cannot match a filter. See [Query performance on lake tables](). - Fine-tuning: further training a model on task examples to change its behaviour. See [Fine-tuning or RAG](). - FinOps: a practice of shared ownership of cloud cost between engineering and finance. See [Cost and FinOps](). - Footer: metadata at the end of the file: schema, chunk positions and statistics. See [Columnar files and Parquet](). - Format version: v2 added row-level deletes; v3 adds deletion vectors and new data types. Engines differ in which versions they support. See [Apache Iceberg: table format basics](). - Forward compatibility: old readers can read new data. See [Schema evolution](). - Framing: turning a question into a precise metric, population, window and decision. See [The data science workflow](). - Freshness: how old the data is when someone reads it. See [Processing patterns: batch, micro-batch and streaming](). - Full compatibility: both directions. See [Schema evolution](). - Full load: copying the complete source dataset on every run. See [Ingestion patterns](). ## G - Generic, singular and unit tests: column checks, custom queries, and logic tests on fixed inputs. See [dbt](). - GitOps: keeping systems in sync with configuration stored in Git. See [Infrastructure as code](). - Gold (curated): tables shaped for a specific use, with owners and stable definitions. See [Medallion layers: bronze, silver and gold](). - Golden record: the combined record after survivorship. See [Multi-source conformance](). - Grain: what exactly one fact row represents. See [Dimensional modelling](). - Grain in the name: naming a table after what one row represents. See [Naming conventions](). - Granularity: how much change one event describes. See [Event design](). - Groundedness: whether an answer is supported by the retrieved sources. See [LLM applications and evaluation](). - Guardrail: a check that blocks or corrects unsafe inputs or outputs. See [LLM applications and evaluation](). ## H - Headless semantic layer: a standalone service exposing metrics through APIs to any tool. See [Semantic layer](). - Hidden partitioning: the table derives partition values from a column by a stored rule. See [Partitioning](). - High-risk AI system: under the EU AI Act, a system in a listed sensitive area with strict requirements. See [Responsible AI and AI governance](). - Hive-style partitioning: partition values written into file paths as key=value. See [Partitioning](). - Home region: the region a record belongs to, decided by a rule such as residence. See [Residency and regions](). - Hub: a table of unique business keys. See [Other modelling approaches: Data Vault, one big table and activity schema](). - Human oversight: people able to review, override and monitor an AI system. See [Responsible AI and AI governance](). - Hybrid search: combining vector similarity with keyword matching. See [Embeddings, vector search and RAG](). - Hypothesis: a candidate explanation that predicts something checkable in the data. See [The data science workflow](). ## I - Iceberg REST catalog: the HTTP API, defined by the Iceberg project, that engines use to talk to a catalog. See [Catalogs](). - Idempotency key: a stable ID used to recognise repeats of the same event. See [Delivery guarantees and idempotency](). - Idempotent model: a model that gives the same result when re-run for the same period. See [Backfill and replay](). - Idempotent operation: one that gives the same result when repeated. See [Delivery guarantees and idempotency](). - Idempotent producer: a Kafka producer whose retries are deduplicated by the broker. See [Delivery guarantees and idempotency](). - Identifier mapping: the field used to match a row to a record in the target. See [Reverse ETL and activation](). - Identity provider: the service that authenticates users and issues tokens. See [Authentication and authorization: OIDC and JWT](). - Identity resolution: assigning one platform ID to all records of one entity. See [Multi-source conformance](). - Incremental load: copying only what changed since the previous run. See [Ingestion patterns](). - Incremental model: a model that processes only new or changed rows on each run. See [dbt](). - Indirect identifier: a value that can identify a person in combination with others. See [PII and data classification](). - Infrastructure as code (IaC): managing infrastructure through declarative, version-controlled files. See [Infrastructure as code](). - Ingestion time: when the event reached the platform. See [Late and out-of-order data](). - Inventory: the list of assets and consumers on the old platform, with usage. See [Platform migration](). - Isolation model: how tenants are separated: shared tables, schema per tenant, or stack per tenant. See [Multi-tenancy](). ## J - JWT: a signed token carrying claims, verified with the issuer's public keys. See [Authentication and authorization: OIDC and JWT](). ## K - Key: the full name of an object within a bucket. See [Object storage and the S3 API](). - Key management service (KMS): a service that holds keys and performs key operations under access control. See [Encryption and key management](). - Key-value store: a database that returns records by key with very low latency. See [Low-latency serving stores](). ## L - Label: the known outcome a model learns to predict. See [Training data management](). - Lakehouse: open files in object storage plus an open table format and a catalog. See [Data lake, warehouse and lakehouse](). - Lance: an open columnar format designed for multimodal data with fast random access. See [Unstructured and multimodal data](). - Landing: writing data to the raw layer as delivered, with load metadata. See [Ingestion patterns](). - Latency: how long processing takes, from an event arriving to its result being available. See [Processing patterns: batch, micro-batch and streaming](). - Lateness: the delay between event time and processing time. See [Late and out-of-order data](). - Lazy evaluation: building a query plan first and running it once, so only needed data is read. See [Python on the lakehouse](). - Leakage: features containing information not available at prediction time. See [Feature engineering and feature stores](). - Least privilege: granting only the access a user or service needs. See [RBAC and ABAC](). - Lineage: the dependencies between datasets, jobs and reports. See [Catalog and lineage](). - Link: a table of relationships between business keys. See [Other modelling approaches: Data Vault, one big table and activity schema](). - Live query: a dashboard query sent to the engine each time it is viewed. See [BI and dashboards](). - LLM-as-a-judge: using a model to score outputs against criteria. See [LLM applications and evaluation](). - LLMOps: practices for building, evaluating and operating LLM applications. See [LLM applications and evaluation](). - Load: writing data into the target store. See [ETL, ELT and EtLT](). - Load metadata: columns added during loading, such as source system and ingestion time. See [ETL, ELT and EtLT](). - Log-based CDC: reading changes from the database's transaction log. See [Change data capture](). - Logical date: the period a run processes, independent of when it runs. See [Orchestration](). - LoRA: a parameter-efficient fine-tuning method that trains small added weight matrices. See [Fine-tuning or RAG](). - LSM tree: a log-structured merge tree, a storage design optimised for frequent writes. See [Table formats compared: Iceberg, Delta Lake, Hudi and Paimon](). ## M - Manifest: a list of files with per-column statistics, used to skip files that cannot match a query. See [Apache Iceberg: table format basics](). - Manifest (Iceberg internals: manifests and planning): an Avro file listing data or delete files with their partition and column metrics. See [Iceberg internals: manifests and planning](). - Manifest list: the per-snapshot list of manifests, with a partition summary for each. See [Iceberg internals: manifests and planning](). - Mapping: aligning fields, types and codes of several sources to one structure. See [Multi-source conformance](). - Materialization: how a model is built: view, table, incremental, ephemeral. See [dbt](). - MCP host: the application in which the agent runs and connects to servers. See [Agents and MCP](). - MCP server: a service that exposes tools over MCP. See [Agents and MCP](). - MERGE: one statement that inserts, updates and deletes rows in a target based on a source. See [Deduplication and merge](). - Metadata column: a column added by the platform, such as `_ingested_at`. See [Naming conventions](). - Metadata file (metadata.json): the root of the tree, holding the schema, partitioning and snapshot list. See [Apache Iceberg: table format basics](). - Metadata table: a read-only table exposing metadata, such as $files or $manifests. See [Iceberg internals: manifests and planning](). - Metric: a named calculation with business meaning, such as revenue. See [Semantic layer](). - Metric, log, trace: numeric measurements over time, event records, and the path of one request or run across components. See [Observability and SLOs](). - Metrics API: an interface that returns metric values by dimension. See [Semantic layer](). - Metrics API (Data APIs and the backend-for-frontend pattern): an interface returning aggregated metric values. See [Data APIs and the backend-for-frontend pattern](). - ML engineer: puts models into production and keeps them running. See [Data team roles](). - MLOps: practices for building, deploying and operating models reliably. See [The ML lifecycle and MLOps](). - Model: one SELECT statement in a file, built into a table or view. See [dbt](). - Model card: documentation of a model's purpose, performance, evaluation and limits. See [Responsible AI and AI governance](). - Model Context Protocol (MCP): an open standard for exposing tools, resources and prompts to agents. See [Agents and MCP](). - Model registry: a catalog of model versions with metadata, status and approvals. See [The ML lifecycle and MLOps](). - Model signature: the declared input and output schema of a model. See [The ML lifecycle and MLOps](). - Module: a reusable package of infrastructure declarations. See [Infrastructure as code](). - MQTT: a light publish and subscribe protocol for devices and unreliable networks. See [IoT and device data](). - Multimodal data: data combining several types, such as images with text. See [Unstructured and multimodal data](). - Multipart upload: uploading a large object in parts that become visible together. See [Object storage and the S3 API](). ## N - Namespace: a grouping of tables, like a schema in a database. See [Catalogs](). - Naming convention: an agreed rule for names, enforced in review or CI. See [Naming conventions](). - Noisy neighbour: a tenant whose load degrades performance for others. See [Multi-tenancy](). - Normalised model: each fact stored once, across many related tables. See [OLTP vs OLAP](). - Notebook: an interactive document of code, results and notes. See [Python on the lakehouse](). ## O - OAuth 2.0: a framework for issuing access tokens to clients. See [Authentication and authorization: OIDC and JWT](). - Object lock: write-once retention that prevents deletion for a period. See [Disaster recovery](). - Observability: understanding a system's state from the signals it emits. See [Observability and SLOs](). - OCR: reading text from images of documents. See [Unstructured and multimodal data](). - ODCS: the Open Data Contract Standard, a YAML format for contracts. See [Data contracts](). - Offline and online store: historical feature values for training, and latest values for low-latency serving. See [Feature engineering and feature stores](). - Offset: a record's position within a partition. See [Kafka and Kafka Connect](). - On-behalf-of identity: the user for whom a service performs an action. See [Audit logging](). - One big table (OBT): a single wide denormalised table for a specific use. See [Other modelling approaches: Data Vault, one big table and activity schema](). - Online serving: scoring per request behind an API, with low latency. See [Model serving and monitoring](). - Open table format: a specification that turns a set of files into a table with transactions and a schema. See [Data lake, warehouse and lakehouse](). - Open table format (Table formats compared: Iceberg, Delta Lake, Hudi and Paimon): a specification that turns files into tables with versions and atomic commits. See [Table formats compared: Iceberg, Delta Lake, Hudi and Paimon](). - Open-weight model: a model whose weights can be downloaded and run on own infrastructure. See [LLM applications and evaluation](). - OpenID Connect: an identity layer on OAuth 2.0 that adds login and ID tokens. See [Authentication and authorization: OIDC and JWT](). - OpenLineage: an open standard for emitting lineage events from running jobs. See [Catalog and lineage](). - Orphan file: a file in the table location that no snapshot references. See [Iceberg maintenance](). - Out-of-core processing: handling data larger than memory by spilling to disk. See [Python on the lakehouse](). - Outbox routing: sending outbox rows to topics by event type or aggregate. See [The outbox pattern](). - Outbox table: a table in the service's database holding events to publish. See [The outbox pattern](). - Output port: the published interface of a data product. See [Data products and data mesh](). ## P - Page: the unit of encoding and compression inside a column chunk. See [Columnar files and Parquet](). - Parallel run: running old and new pipelines side by side and comparing outputs. See [Platform migration](). - Parsing: extracting text and structure from documents. See [Unstructured and multimodal data](). - Partition: an ordered, append-only part of a topic; the unit of ordering and parallelism. See [Kafka and Kafka Connect](). - Partition key: the column, or derived value, that decides which partition a row belongs to. See [Partitioning](). - Partition pruning: skipping partitions that cannot match a query's filter. See [Partitioning](). - Partition summary: the lowest and highest partition values in one manifest. See [Iceberg internals: manifests and planning](). - Partition transform: the rule that derives partition values, such as day(ordered\_at) or bucket(16, customer\_id). See [Apache Iceberg: table format basics](). - Partition-scoped run: a run that processes and writes exactly one period. See [Backfill and replay](). - Peeking: repeatedly checking results and stopping early. See [Experimentation and A/B testing](). - Personal data (PII): information relating to an identified or identifiable person. See [PII and data classification](). - Physical deletion: the data no longer exists in any file, as opposed to being hidden from queries. See [Privacy lifecycle](). - Plan: a preview of the changes an apply would make. See [Infrastructure as code](). - Platform engineer: builds and operates the shared data platform. See [Data team roles](). - Platform team: provides shared tooling and enforces cross-cutting rules. See [Governance roles](). - Point-in-time join: joining on a key and a time within the validity range. See [History and slowly changing dimensions](). - Point-in-time join (Feature engineering and feature stores): matching each example with feature values as they were at its time. See [Feature engineering and feature stores](). - Policy as code: access and handling rules written in files, often in a policy language, and tested. See [Governance as code](). - Policy decision point: the component that evaluates access rules. See [RBAC and ABAC](). - Policy enforcement point: the component that applies the decision to a request. See [RBAC and ABAC](). - Policy engine: a service that evaluates policies at request time, such as Open Policy Agent. See [Governance as code](). - Power: the probability of detecting an effect of a given size if it exists. See [Experimentation and A/B testing](). - Pre-aggregation: a stored summary table used to answer common queries quickly. See [Semantic layer](). - Pre-signed URL: a time-limited link that grants read access to one object. See [Data sharing](). - Predicate pushdown: using a filter to skip data before reading it. See [Columnar files and Parquet](). - Predicate pushdown (Query performance on lake tables): passing a filter to the table reader so data can be skipped before it is read. See [Query performance on lake tables](). - Prediction log: a record of every prediction with inputs and model version. See [Model serving and monitoring](). - Prefix: the start of a key, used for listing and access rules; what consoles show as a folder. See [Object storage and the S3 API](). - Prefix (Naming conventions): a name start that marks a layer or table type, such as clean\_ or fct\_. See [Naming conventions](). - Primary and guardrail metrics: what should improve, and what must not get worse. See [Experimentation and A/B testing](). - Probabilistic matching: matching on scored similarity across several fields. See [Multi-source conformance](). - Processing time: when the pipeline handles the event. See [Late and out-of-order data](). - Producer: the team that creates and changes the dataset. See [Data contracts](). - Projection: reading only the columns a query needs. See [Columnar files and Parquet](). - Projection pushdown: reading only the columns a query uses. See [Query performance on lake tables](). - Promotion: moving a model version to a higher status, such as production, under defined rules. See [The ML lifecycle and MLOps](). - Prompt injection: instructions hidden in data that try to steer the model. See [Agents and MCP](). - Prompting: guiding a model with instructions and examples in each request. See [Fine-tuning or RAG](). - Provider: the party that owns and shares the data. See [Data sharing](). - Pseudonymisation: replacing identifiers with pseudonyms while the re-identification key is kept apart. See [PII and data classification](). - Puffin file: a file of extra table statistics, such as distinct value estimates, or deletion vectors. See [Iceberg internals: manifests and planning](). - Pull and push: the platform fetches data, or the source sends it. See [Ingestion patterns](). - Pull request: a proposed change, reviewed and checked before merging. See [Environments and CI](). - PyIceberg: the Python library for reading and writing Iceberg tables. See [Python on the lakehouse](). ## Q - QoS level: MQTT's delivery guarantee per message: 0, 1 or 2. See [IoT and device data](). - Query plan: the steps an engine will run, shown by EXPLAIN. See [Query performance on lake tables](). - Query-based CDC: finding changes by querying a change column on a schedule. See [Change data capture](). ## R - RACI: a matrix of who is Responsible, Accountable, Consulted and Informed per decision. See [Governance roles](). - Randomisation unit: what is assigned to a group, such as a user. See [Experimentation and A/B testing](). - Range read: reading only part of an object by byte offsets. See [Object storage and the S3 API](). - Rate limit: the maximum number of API calls a target allows per period. See [Reverse ETL and activation](). - Raw layer: the copy of source data kept as delivered, the basis for re-running transformations. See [ETL, ELT and EtLT](). - RBAC: permissions granted to roles; users get permissions through roles. See [RBAC and ABAC](). - Real-time OLAP: databases that answer aggregate queries in milliseconds on fresh data. See [Low-latency serving stores](). - ReBAC: permissions decided from relationships between users and objects. See [RBAC and ABAC](). - Rebuild: recreating a layer from the one below after a logic change or a bug. See [Medallion layers: bronze, silver and gold](). - Recipient: the party that reads shared data with its own tools. See [Data sharing](). - Reconciliation: comparing outputs to find and explain differences. See [Platform migration](). - Reconciliation test: compares counts or totals across tables or systems. See [Testing data pipelines](). - Recovery point objective (RPO): the maximum acceptable data loss, as time. See [Disaster recovery](). - Recovery time objective (RTO): the maximum acceptable time to restore. See [Disaster recovery](). - ref(): a reference to another model, from which dbt builds the dependency graph. See [dbt](). - Regional stack: a full set of storage, catalog and compute for one region. See [Residency and regions](). - Relationship: how tables join, declared once in the layer. See [Semantic layer](). - Remote signing: the catalog signing each storage request on the engine's behalf. See [Catalogs](). - Replay: reprocessing data from the raw layer or from a log. See [Backfill and replay](). - Replication slot: PostgreSQL's record of how far a consumer has read the log. See [Change data capture](). - Reprocessing window: the number of recent days a batch job recomputes on each run. See [Late and out-of-order data](). - Reproducibility: the ability to rerun an analysis on the same data and code and get the same result. See [The data science workflow](). - Resource group: an engine setting that limits resources for a set of queries. See [Multi-tenancy](). - Resource tagging: labelling cloud resources with owner and purpose. See [Cost and FinOps](). - Retained message: the last message on a topic, kept by the broker for new subscribers. See [IoT and device data](). - Retention: how long or how much data a topic keeps. See [Kafka and Kafka Connect](). - Retention period: how long snapshots are kept, which is also how far back time travel reaches. See [Iceberg maintenance](). - Retention period (Privacy lifecycle): how long a kind of data is kept, and why. See [Privacy lifecycle](). - Retrieval-augmented generation (RAG): answering with a model that is given retrieved documents. See [Embeddings, vector search and RAG](). - Retrieval-augmented generation (RAG) (Fine-tuning or RAG): adding retrieved documents to each request. See [Fine-tuning or RAG](). - Retry: re-running a failed task automatically after a delay. See [Orchestration](). - Reverse ETL: syncing platform results into operational tools. See [Reverse ETL and activation](). - Rollback: making an earlier snapshot the current state of a table. See [Disaster recovery](). - Row filter: a condition that limits which rows a user sees. See [Row-level security and masking](). - Row group: a horizontal slice of a file, holding one column chunk per column. See [Columnar files and Parquet](). - Row-oriented storage: all columns of a row stored together. See [OLTP vs OLAP](). - Run parameter: the period a run processes, passed in instead of taken from the clock. See [Backfill and replay](). - Runbook: written steps for responding to an alert. See [Observability and SLOs](). ## S - Sample ratio mismatch (SRM): a difference between planned and actual group sizes that signals a bug. See [Experimentation and A/B testing](). - Sandbox schema: a personal or project area for experimental tables. See [The data science workflow](). - Satellite: a table of descriptive attributes and their history, attached to a hub or link. See [Other modelling approaches: Data Vault, one big table and activity schema](). - Scan: reading a large part of a table rather than looking up single rows. See [OLTP vs OLAP](). - Scheduler: the component that starts runs and launches tasks when ready. See [Orchestration](). - Schema: the declared names, types and constraints of a table's columns. See [Schema-on-write and schema-on-read](). - Schema drift: unannounced changes in the structure of source data. See [Schema-on-write and schema-on-read](). - Schema enforcement: rejecting writes that do not match the schema. See [Schema-on-write and schema-on-read](). - Schema registry: a service that stores schema versions and checks compatibility. See [Schema evolution](). - Schema test: checks columns, types and constraints. See [Testing data pipelines](). - Secrets manager: a service that stores and distributes credentials securely. See [Encryption and key management](). - Self-serve platform: shared infrastructure that lets teams build and publish products on their own. See [Data products and data mesh](). - Self-service analytics: users exploring governed data on their own. See [BI and dashboards](). - Semi-structured data: records with structure that may vary, such as JSON. See [Schema-on-write and schema-on-read](). - Sensor: a task that waits for a condition before downstream tasks start. See [Orchestration](). - Separation of storage and compute: data stored once, processed by engines that can be scaled or swapped independently. See [Data lake, warehouse and lakehouse](). - Sequence number: the order of commits, used to decide which deletes apply to which files. See [Iceberg internals: manifests and planning](). - Service account: an identity for a non-human client. See [Authentication and authorization: OIDC and JWT](). - Service level: a promise about availability, such as data ready by 07:00. See [Data contracts](). - Service level agreement (SLA): a promise to customers, with consequences if broken. See [Observability and SLOs](). - Service level indicator (SLI): the measured quantity, such as completion time. See [Observability and SLOs](). - Service level objective (SLO): a target for an SLI over a period, such as 98% of days. See [Observability and SLOs](). - Serving store: a database optimised for fast application queries, fed by the platform. See [Data APIs and the backend-for-frontend pattern](), [Low-latency serving stores](). - Shadow deployment: running a new model on live traffic without using its predictions. See [Model serving and monitoring](). - Share: a named set of tables or views made available to a recipient. See [Data sharing](). - Sharing protocol: an open API for discovering and reading shared tables. See [Data sharing](). - Showback and chargeback: reporting cost to teams, or billing it to their budgets. See [Cost and FinOps](). - SIEM: a system that collects security logs and detects suspicious patterns. See [Audit logging](). - Silver (clean): typed, deduplicated, conformed entities. See [Medallion layers: bronze, silver and gold](). - Sink connector: a Kafka Connect component that writes topic records to an outside system. See [Writing to Iceberg from Kafka](). - Slim CI: building only changed models and their dependants, reusing production for the rest. See [Environments and CI](). - Slowly changing dimension: a dimension whose attributes change over time. See [History and slowly changing dimensions](). - Snapshot: the state of the table after one commit. See [Apache Iceberg: table format basics](). - Snapshot (Change data capture): the initial read of a table's current rows before streaming changes (not an Iceberg snapshot). See [Change data capture](). - Snapshot expiry: removing old snapshots and the files only they referenced. See [Iceberg maintenance](). - Snapshot table: a full copy of a table per day or period. See [History and slowly changing dimensions](). - Soft delete: marking a row as deleted instead of removing it; it keeps the stored sequence, so a replay cannot bring a deleted row back. See [Deduplication and merge](). - Source: a raw table loaded outside dbt, declared so models can reference it. See [dbt](). - Source of truth: the system whose data is authoritative when copies disagree. See [Low-latency serving stores](). - Source sequence: an ordering from the source, such as a log position or offset. See [Deduplication and merge](). - Star schema: a fact table joined directly to its dimensions. See [Dimensional modelling](). - State: what a stream processor remembers between events, such as running totals. See [Processing patterns: batch, micro-batch and streaming](). - State (Infrastructure as code): the tool's record mapping declarations to real resources. See [Infrastructure as code](). - State (Stream processing): data a job keeps between events, partitioned by key. See [Stream processing](). - Storage limitation: the principle of keeping personal data no longer than needed. See [Privacy lifecycle](). - Stream processor: an engine that runs continuous jobs over event streams. See [Stream processing](). - Stream-table join: enriching events with the current version of a table held in state. See [Stream processing](). - Subject: the business object an event is about. See [Event design](). - Suffix: a name end that marks a type or meaning, such as `_at` or `_key`. See [Naming conventions](). - Suppression list: IDs of erased people, used to block reintroduction. See [Privacy lifecycle](). - Surrogate key: a key created in the warehouse to identify a dimension row. See [Dimensional modelling](). - Survivorship: per-attribute rules for which source value wins. See [Multi-source conformance](). - Sync: a configured transfer from a source model to a destination object. See [Reverse ETL and activation](). - Synthetic data: generated data that imitates real data. See [Training data management](). ## T - Table statistics: row counts and distinct value estimates used by the optimizer. See [Query performance on lake tables](). - Tag: a named reference to one snapshot, with its own retention. See [Write-audit-publish](). - Target: the configuration that decides where and how a run writes. See [Environments and CI](). - Task: one unit of work, such as running a model or loading a file. See [Orchestration](). - Technical catalog: maps table names to current metadata and coordinates commits. See [Catalogs](). - Technical metadata: structure, such as tables, columns and types. See [Catalog and lineage](). - Telemetry: periodic measurements sent by devices. See [IoT and device data](). - Tenant: a customer or organisation whose data must be kept separate. See [Multi-tenancy](). - Tenant key: the column identifying which tenant a row belongs to. See [Multi-tenancy](). - Test fixture: the fixed input rows used by a unit test. See [Testing data pipelines](). - Tie-breaker: an extra ordering column that makes the choice deterministic. See [Deduplication and merge](). - Time-based split: splitting by date so the model is tested on later data than it learned from. See [Training data management](). - Timeline: Hudi's record of actions on a table. See [Table formats compared: Iceberg, Delta Lake, Hudi and Paimon](). - Timeliness (freshness): availability by the time it is needed. See [Data quality dimensions](). - Tokenisation: replacing a sensitive value with a token that can be reversed only by a separate, protected service. See [Row-level security and masking](). - Tombstone: a record with a key and no value, marking a deleted key. See [Change data capture](). - Tool: a function an agent can call, described by name, purpose and input schema. See [Agents and MCP](). - Topic: a named stream of records. See [Kafka and Kafka Connect](). - Topic hierarchy: slash-separated topic names that can be matched with + and \# wildcards. See [IoT and device data](). - Trace: the full record of one request through an LLM application. See [LLM applications and evaluation](). - Training run: one execution of training on a specific data version, code version and settings. See [The ML lifecycle and MLOps](). - Training, validation and test sets: data for learning, for tuning, and for final evaluation. See [Training data management](). - Training-serving skew: a difference between features at training and at prediction time. See [Feature engineering and feature stores](). - Transaction: a group of reads and writes that succeeds or fails as a whole. See [OLTP vs OLAP](). - Transaction log: Delta Lake's ordered record of commits. See [Table formats compared: Iceberg, Delta Lake, Hudi and Paimon](). - Transcription: converting speech to text. See [Unstructured and multimodal data](). - Transform: cleaning, combining and reshaping data for use. See [ETL, ELT and EtLT](). - Type 1: overwrite; no history. See [History and slowly changing dimensions](). - Type 2: one row per version, with validity dates. See [History and slowly changing dimensions](). ## U - Unbounded data: a stream with no end. See [Processing patterns: batch, micro-batch and streaming](). - UniForm: Delta Lake's feature that also writes Iceberg-compatible metadata. See [Table formats compared: Iceberg, Delta Lake, Hudi and Paimon](). - Uniqueness: no duplicate entities. See [Data quality dimensions](). - Unit cost: cost per unit of value, such as per dashboard view or per million orders processed. See [Cost and FinOps](). - Unit test: runs logic on fixed inputs and compares with expected output. See [Testing data pipelines](). - Unstructured data: content without a fixed tabular structure, such as documents, images and audio. See [Unstructured and multimodal data](). - Upsert mode: records replace or delete existing rows by key. See [Writing to Iceberg from Kafka](). ## V - Validity: conformance to type, format and allowed values. See [Data quality dimensions](). - Versioned load: loading a new table version and switching readers to it at once. See [Low-latency serving stores](). ## W - Watermark: the stored position, such as the highest updated\_at loaded, from which the next run continues (not the streaming watermark in [Late and out-of-order data]()). See [Ingestion patterns](). - Watermark (streaming): an estimate that no events older than a given time are still to come. See [Late and out-of-order data](). - Window: a slice of a stream, such as the last five minutes, used to compute results. See [Processing patterns: batch, micro-batch and streaming](). - Window (Stream processing): a time-based grouping of events: tumbling, sliding or session. See [Stream processing](). - Winning record: the record chosen as current for a key. See [Deduplication and merge](). - Worker: a process that reads data and executes part of the plan. See [SQL query engines](). - Write-audit-publish (WAP): write unseen, check, then publish atomically. See [Write-audit-publish](). - Write-once storage: storage where records cannot be changed or deleted for a retention period. See [Audit logging]().