Encryption and key management¶

Protecting data by making it unreadable without a key, both while it moves between systems and while it is stored, and managing those keys so that the right systems can use them and nobody else can.

Before reading

Object storage and the S3 API · Privacy lifecycle

Why it matters¶

Access control decides who may read data through the platform's front doors. Encryption protects it when those controls are bypassed: a stolen disk, a misconfigured bucket, a copied backup, a network eavesdropper. Most security standards and contracts require it, and key management makes it possible to delete data that cannot otherwise be reached.

How it works¶

Encryption in transit protects data moving over networks. Every connection, such as an engine to object storage or a client to Kafka, uses TLS, which also verifies the server's identity with a certificate. Internal traffic too: "inside the network" is not a security boundary.

Encryption at rest protects stored data. Storage encrypts data with a key before writing it and decrypts it for authorised reads.

Server-side encryption is done by the storage service. Provider-managed keys protect against physical theft, but anyone who can read through the service sees plaintext. With customer-managed keys in a key management service (KMS), the organisation controls the key: it decides which identities may use it, sees every use in audit logs, and can disable it, which makes the data unreadable.

Client-side encryption happens before data reaches storage, so the storage service never sees plaintext. Parquet modular encryption encrypts individual columns of a file with different keys, so a sensitive column can be readable only to engines holding its key. Iceberg table encryption builds on this: data, delete, manifest and manifest-list files are encrypted (metadata.json is not); each file's key is stored encrypted in its parent manifest, up a chain that ends in a table master key held in the KMS. As of Iceberg 1.12 (checked October 2026) it works with the Hive and REST catalogs for Parquet and Avro tables.

Envelope encryption is how this scales. Data is encrypted with a data encryption key (DEK), generated per file or object. The DEK is encrypted ("wrapped") with a key encryption key (KEK) that never leaves the KMS, and stored alongside the data. To read, a service asks the KMS to unwrap the DEK, which it does only for authorised callers, then decrypts locally.

flowchart LR
  D[Data file] -->|encrypt with| DEK[Data key,<br/>one per file]
  DEK -->|wrapped by| KEK[Key encryption key<br/>in the KMS]
  W[Wrapped data key<br/>stored with the file] -.-> D
  R[Engine reading] -->|unwrap request,<br/>if authorised| KMS[KMS:<br/>logs every use]
  KMS --> R

Key rotation replaces keys periodically or after an incident. With envelope encryption, rotating the KEK needs no re-encryption of data: at most the small DEKs are re-wrapped, and managed KMS rotation often keeps old key versions for decryption, so not even that.

Key scope is a design decision: separate keys per environment, region or tenant mean that disabling one key affects only that scope, and a region's data can use a key held in that region. Crypto-shredding uses the smallest scope: one key per person for their identifying fields, so erasing the key erases the data everywhere, including old snapshots and backups.

Secrets, such as database passwords and client secrets, are keys too. They live in a secrets manager, are injected into jobs at run time, rotate regularly, and never appear in code, notebooks or logs.

Key terms¶

Encryption in transit: protecting data on the network, usually with TLS.
Encryption at rest: protecting stored data.
Key management service (KMS): a service that holds keys and performs key operations under access control.
Envelope encryption: encrypting data with data keys that are themselves encrypted by a master key.
Customer-managed key: an encryption key controlled by the organisation, not the provider.
Secrets manager: a service that stores and distributes credentials securely.

Example¶

The shop's EU customer tables use a customer-managed key in the EU region's KMS:

The bucket uses server-side encryption with that key; only the engine and pipeline roles of the EU stack may use it, and every use is logged.
The street and postcode columns of clean_customers, masked for every query user, are also encrypted with Parquet column keys that only the ingestion job and the privacy team's jobs can unwrap. The shared engine never receives these keys, so a misconfigured masking policy cannot expose the values. Email stays plain, because support and the CRM sync read it through the engine.
The catalog's database password and the CDC connector's credentials rotate every 90 days in the secrets manager.

Asked who could read EU customer data last quarter, the shop finds that the key policy shows who could use the key, and the KMS log shows which roles did; with bucket-level keys, per-object reads appear only in the storage access logs.

Pitfalls¶

Treating encryption at rest as access control. Anyone who can read through the service sees plaintext.
One key for everything. Disabling it affects the whole platform.
Keys and data held by the same people. Separate key managers from data readers.
Secrets in notebooks and repositories. They leak through history, screenshots and copies.
Untested key loss. A deleted or disabled key makes data unrecoverable; protect keys from deletion and test recovery.

Learn more¶

Apache Iceberg, encryption: https://iceberg.apache.org/docs/latest/encryption/
Apache Parquet, modular encryption: https://parquet.apache.org/docs/file-format/data-pages/encryption/
NIST SP 800-57 Part 1, recommendation for key management: https://csrc.nist.gov/pubs/sp/800/57/pt1/r5/final

Audit logging · Residency and regions · Multi-tenancy · Catalogs