Skip to content

Iceberg maintenance

Regular jobs that keep an Iceberg table fast and its storage under control: compacting small files, expiring old snapshots and removing files that nothing references.

Before reading

Apache Iceberg: table format basics

Why it matters

Every commit adds files and keeps the old ones, so that earlier snapshots stay readable. A table written by a stream gains thousands of small files and delete files a day. Without maintenance, queries slow down, planning slows down, storage grows without limit, and deleted rows stay in storage.

How it works

Four jobs do most of the work.

Compaction rewrites many small data files into fewer large ones, up to the table's target size (512 MB by default in Iceberg's own procedures; engines may differ). It also applies delete files while rewriting, so readers no longer have to merge them at query time. The simplest strategy, bin-packing, only combines files. A sort strategy also reorders rows, for example by customer_id, which makes file statistics more selective. Compaction is an ordinary commit that creates a new snapshot. The old small files are not deleted yet, because earlier snapshots still reference them.

Rewriting manifests regroups the table's manifest files. Many small commits create many small manifests, and planning has to read all of them; grouping manifests by partition lets planning skip most of them.

Expiring snapshots removes snapshots older than a retention period, always keeping at least the latest one. Files referenced only by expired snapshots, such as the small files replaced by compaction, are then deleted. Time travel to those snapshots is no longer possible. Old metadata.json files are deleted only if the table enables write.metadata.delete-after-commit.enabled, which is off by default; write.metadata.previous-versions-max, 100 by default, sets how many are kept.

Removing orphan files deletes files in the table's location that no snapshot references. Orphans come from failed or interrupted writes. The job lists storage and compares it with the metadata, so it must ignore recent files: a write still in progress has files that look like orphans until it commits. Only files older than a safety window, three days or more, are removed.

flowchart LR
  W[Streaming writer<br/>commits every 5 minutes] --> T[(shop.raw_orders:<br/>small files,<br/>old snapshots)]
  T --> C["1. Compact<br/>small files into large;<br/>old files still referenced<br/>by earlier snapshots"]
  C --> M["2. Rewrite manifests<br/>grouped by partition"]
  M --> E["3. Expire snapshots<br/>older than retention;<br/>files only they used<br/>are deleted"]
  E --> O["4. Remove orphan files<br/>older than safety window;<br/>files never committed<br/>are deleted"]

The order matters. Compaction first, so the files it replaces become unreferenced, then rewriting manifests. Expiry next, so the replaced files are deleted. Orphan removal last and least often, as a safety net.

Frequency follows the write rate. A table streamed every few minutes may need compaction of recent partitions every hour; a table loaded nightly may need it weekly. Expiry usually runs daily and orphan removal weekly. Compaction should target partitions that changed since the last run, not the whole table.

Compaction can conflict with other writers. Plain appends do not conflict with it; concurrent updates, deletes or upserts on the same files make one of the commits fail and retry. Schedule compaction away from large batch updates.

Format version 3 replaces position delete files with deletion vectors, one compact bitmap per data file, which keeps merge-on-read tables cheaper to read between compactions.

Key terms

  • Compaction: rewriting small files, and their delete files, into fewer large files.
  • Bin-packing: compaction that combines files without reordering rows.
  • Snapshot expiry: removing old snapshots and the files only they referenced.
  • Retention period: how long snapshots are kept, which is also how far back time travel reaches.
  • Orphan file: a file in the table location that no snapshot references.
  • Deletion vector: a bitmap marking deleted rows of one data file, in format version 3.

Example

The shop's raw_orders table, partitioned by day(_ingested_at), is fed by the Kafka Connect sink with four tasks, committing every five minutes: about 288 commits and 1,150 small files a day. A maintenance job in Trino:

-- how many files, how small
SELECT count(*) AS files, avg(file_size_in_bytes) / 1e6 AS avg_mb
FROM shop."raw_orders$files";

-- merge files smaller than 100 MB, in yesterday's partition only
-- (a range on the partition column, aligned to the day, so whole partitions match)
ALTER TABLE shop.raw_orders EXECUTE optimize(file_size_threshold => '100MB')
WHERE _ingested_at >= TIMESTAMP '2026-10-28 00:00:00 UTC'
  AND _ingested_at < TIMESTAMP '2026-10-29 00:00:00 UTC';

-- drop snapshots older than 7 days and the files only they used
ALTER TABLE shop.raw_orders EXECUTE expire_snapshots(retention_threshold => '7d');

-- delete files never committed, older than 7 days
ALTER TABLE shop.raw_orders EXECUTE remove_orphan_files(retention_threshold => '7d');

Before the job, a day's partition holds about 1,150 files averaging about 600 KB. After it, the same day is a handful of files, and a week later, once the old snapshots expire, the small files are gone from storage.

Pitfalls

  • Short snapshot retention. It removes time travel and can break long-running queries that are still reading an old snapshot.
  • A short orphan safety window. Files of a write in progress are deleted and the table is corrupted.
  • Deleted rows are not erased until compaction rewrites the data files that hold them and expiry then removes the old files; see Privacy lifecycle.
  • Compacting the whole table every run. Cost grows with table size while only recent partitions changed.
  • No owner. Maintenance is easy to forget until a table is slow; schedule and monitor it like any pipeline.

Learn more

  • Apache Iceberg, maintenance: https://iceberg.apache.org/docs/latest/maintenance/
  • Apache Iceberg, Spark procedures: https://iceberg.apache.org/docs/latest/spark-procedures/
  • Trino Iceberg connector, table procedures: https://trino.io/docs/current/connector/iceberg.html

Iceberg internals: manifests and planning · Query performance on lake tables · Privacy lifecycle · Writing to Iceberg from Kafka · Orchestration