Skip to content

Governance as code#

Optional

Keeping governance decisions, such as owners, classification tags, glossary links, access grants and policies, in version-controlled files, and having automation apply and check them, instead of setting them by hand in each tool.

Before reading

Catalog and lineage · Data contracts

Why it matters#

Governance set by hand drifts. Someone tags a column as personal data in the catalog, but the masking rule in the engine is configured separately and forgotten. Access granted for a project is never removed. Nobody can say who changed what, or why. Treating governance like code gives it review, history, testing and repeatable deployment, and lets one decision drive every system that must act on it.

How it works#

The idea is to declare once and apply everywhere. Governance facts are written in files that live with the data's code: contract files, dbt YAML, or dedicated policy files. They cover:

  • ownership: owner, steward and team for each dataset;
  • classification: sensitivity tags per column, such as pii or confidential;
  • business terms: links from columns to glossary entries;
  • access: which roles or attributes may read which datasets;
  • policies: rules such as "columns tagged pii are masked except for roles with a privacy approval".

Changes go through pull requests. A steward proposes tagging a new column; the owner approves in review; the history records who decided and when.

On merge, continuous integration applies the declared state to each system through its API: tags and owners to the catalog, grants to the engine or technical catalog, policy definitions to the policy engine. Tools that manage infrastructure declaratively, such as Terraform, are often used for grants and roles. Policies themselves can be written in a policy language, such as Rego for Open Policy Agent, and tested like any other code.

flowchart LR
  F[Files in Git:<br/>contracts, tags,<br/>owners, policies] -->|pull request<br/>and review| CI[CI pipeline]
  CI -->|checks: every published<br/>table has owner and tags| CI
  CI -->|tags, owners, terms| CAT[Business catalog]
  CI -->|grants| ENG[Engine and<br/>technical catalog]
  CI -->|masking and row policies| POL[Policy engine]

CI also checks rules about governance itself: every published table has an owner, every column in a gold table has a classification, no column that looks like an email address is untagged, no access grant goes to an individual instead of a role. A pull request that breaks these rules fails, like one that breaks a test.

Drift detection closes the loop. A scheduled job compares what is declared with what is actually configured in each system and reports differences, such as a grant someone added by hand. Either the change is brought into the files, or it is removed.

The approach does not replace people's decisions. It records them, routes them through review, and makes sure they are applied consistently.

Key terms#

  • Declarative configuration Describing the desired state, and letting tools make the systems match it.
  • Policy as code Access and handling rules written in files, often in a policy language, and tested.
  • Classification tag A label on a column or table marking its sensitivity.
  • Drift A difference between declared and actual configuration.
  • Policy engine A service that evaluates policies at request time, such as Open Policy Agent.

Example#

Tagging PII columns from contract files. The customers contract marks sensitive fields:

schema:
  - name: clean_customers
    properties:
      - name: email
        logicalType: string
        classification: confidential
        tags: [pii, contact]
      - name: country
        logicalType: string
        classification: internal

On merge, a CI step reads every contract and pushes the classifications to the catalog as tags and to the policy engine as data. The masking policy, in Rego for Trino's Open Policy Agent integration, is written once, against the tag rather than the column:

package trino

# column_tags is pushed by CI from the contract files,

<span class="wiki-badge">Optional</span>
# keyed by catalog, schema, table and column

<span class="wiki-badge">Optional</span>
columnMask := {"expression": "NULL"} if {
  c := input.action.resource.column
  "pii" in data.column_tags[c.catalogName][c.schemaName][c.tableName][c.columnName]
  not "privacy-approved" in input.context.identity.groups
}

Support, privacy and the CRM sync service are members of privacy-approved, so they see email; everyone else gets null. When the shop adds a mobile_phone column tagged pii, it is masked from the moment the contract is merged, without anyone changing the policy.

Pitfalls#

  • Two sources of truth. If tags can still be edited in the catalog UI, the files and the catalog diverge; make one authoritative.
  • Applying without checking. CI that pushes changes but never validates rules gives speed without safety.
  • No drift detection. Manual changes accumulate unseen.
  • Policies tied to column names; see Row-level security and masking.
  • Everything as code at once. Start with owners and classification, then add grants and policies.

Learn more#

PII and data classification Row-level security and masking Environments and CI Naming conventions