# Data quality dimensions The aspects in which data can be good or bad, such as accuracy, completeness and timeliness, and how each is measured and watched in production. > [!NOTE] > > **Before reading** > > [Testing data pipelines]() ## Why it matters "The data is wrong" is not actionable. "Yesterday's orders are 30% incomplete" is. Quality dimensions give a shared vocabulary for what can go wrong, make quality measurable, and let owners set targets for what matters to their consumers instead of chasing perfection everywhere. ## How it works Six dimensions are commonly used. Accuracy: the data matches reality. An order amount equals what the customer paid. Accuracy is hard to measure directly; it is usually checked by comparing with a trusted source, such as payments against the payment provider's settlement report. Completeness: everything that should be there is there. All orders of the day were loaded, and required fields are filled. Measured as row counts against the source or an expected volume, and as the share of nulls in required columns. Consistency: the same fact agrees across places. Revenue in fct\_daily\_sales equals the sum of order line amounts in clean\_order\_items for orders that are not cancelled; an order's customer exists in dim\_customer. Measured with reconciliation and relationship checks. Timeliness, or freshness: data is available when it is needed. Measured as the age of the newest data, for example the time since the latest ordered\_at, or as whether the table was updated by its agreed time. Validity: values follow the rules of their type and domain. Dates are real dates, statuses come from the known list, email addresses have a valid form. Measured with schema and data tests. Uniqueness: each entity appears once. Measured as duplicate counts on keys. Tests run before publishing check rules on one run's output. Monitoring watches the same dimensions over time, on published data, and finds problems no single rule describes: a table that suddenly has 40% fewer rows than the same weekday last week, a column whose null rate jumps from 1% to 15%, a distribution that shifts. These anomaly checks compare against history rather than a fixed rule. ``` flowchart LR T[(clean_orders)] --> M[Checks after each run:
freshness: newest order under 2 hours old
volume: within 25% of same weekday
nulls in customer_id under 0.1%] M --> D{All pass?} D -->|yes| OK[Status: healthy] D -->|no| AL[Alert custodian or steward,
mark table degraded
in the catalog] ``` Each check needs a threshold and an owner. Thresholds come from what consumers need, written in the contract, not from what the data happens to look like. Alerts go to the people who can act, usually the custodian for loading problems and the steward for content problems. Every alert that fires without anyone acting is a sign that the threshold or the routing is wrong. Quality status should be visible to consumers: the catalog can show the last check results and whether a table is currently degraded, so analysts do not unknowingly build on bad data. ## Key terms - Accuracy: agreement with reality or a trusted source. - Completeness: no missing records or required values. - Consistency: agreement of the same fact across tables and systems. - Timeliness (freshness): availability by the time it is needed. - Validity: conformance to type, format and allowed values. - Uniqueness: no duplicate entities. - Anomaly detection: flagging values that differ from their usual pattern. ## Example Freshness and completeness checks on clean\_orders, run every hour: ``` -- freshness: minutes since the newest order SELECT date_diff('minute', max(ordered_at), current_timestamp) AS minutes_behind FROM shop.clean_orders; -- alert if above 120 -- completeness: today's orders against the same weekday last week, same hour SELECT count_if(ordered_at >= current_date) AS today, count_if(ordered_at >= current_date - INTERVAL '7' DAY AND ordered_at < current_timestamp - INTERVAL '7' DAY) AS last_week FROM shop.clean_orders WHERE ordered_at >= current_date - INTERVAL '7' DAY; -- alert if today is more than 25% below last_week ``` On a Tuesday at 11:00, minutes\_behind is 190 and today is 60% of last week. The custodian finds the CDC connector stopped at 07:50; the catalog shows clean\_orders as degraded until it catches up. ## Pitfalls - Thresholds set by guess. Base them on consumer needs and on the data's real variation. - Alerts to a shared channel nobody owns. Route them to named people. - Measuring everything equally. Focus on the tables and dimensions consumers depend on. - Fixed thresholds on seasonal data. Weekends, holidays and sales distort volume; compare with the same weekday or use anomaly checks. - Quality status hidden from consumers. They keep using degraded data without knowing. ## Learn more - DAMA International, DAMA-DMBOK, 2nd edition (Technics Publications, 2017), chapter 13, data quality. - Barr Moses, Lior Gavish and Molly Vorwerck, Data Quality Fundamentals (O'Reilly, 2022). ## Related [Observability and SLOs]() · [Data contracts]() · [Write-audit-publish]() · [Governance roles]()