# Disaster recovery Optional Being able to restore the platform's data and service after a serious failure, such as a deleted table, corrupted data, a lost catalog or an unavailable region, within agreed limits on lost data and downtime. > [!NOTE] > > **Before reading** > > [Catalogs]() · [Iceberg maintenance]() ## Why it matters Object storage is very durable, which makes it easy to assume data cannot be lost. Most losses are not hardware failures: someone drops a table, a job overwrites a partition with bad data, a lifecycle rule deletes files, credentials are misused, or a region goes down. Recovery that has never been planned or tested usually takes far longer than anyone expected, or fails. ## How it works Two numbers frame the plan for each dataset or service. The recovery point objective (RPO) is how much data may be lost, measured in time: an RPO of one hour means restoring to a state at most an hour old. The recovery time objective (RTO) is how long the restore may take. Tighter numbers cost more, so they are set per dataset by its owner: published finance tables may need an RTO of hours and an RPO of one day; a sandbox may need no recovery at all. A lakehouse has several things to protect, and each needs its own approach. Data files in object storage. Bucket versioning keeps overwritten and deleted objects for a period, protecting against accidental deletion. Object lock, a write-once retention setting, protects backups against deletion, in compliance mode even by administrators or attackers. Replication to another bucket, often in another region, protects against regional loss. Table metadata. Iceberg's own history helps with logical mistakes: snapshots and time travel let a table be rolled back to an earlier snapshot, as long as that snapshot has not expired. This is often the fastest recovery, measured in minutes. The catalog. It holds the pointer to each table's current metadata and the access rules. It is usually a database, and must be backed up like one. If it is lost, tables can be registered again from their latest metadata files, but permissions and other catalog state are lost unless backed up. Code and configuration: pipelines, models, contracts and infrastructure definitions live in version control, from which the platform can be redeployed. The raw layer is the last line of defence for derived data. If clean and gold tables are lost or corrupted, they can be rebuilt from raw with the same code, as in a backfill. This only works if raw data itself is protected and kept long enough. ``` flowchart TB I[Incident] --> Q{What was lost?} Q -->|bad write,
earlier snapshot still exists| R1[Roll back to an
earlier snapshot: minutes] Q -->|table dropped,
files still in bucket| R2[Register table again
from its metadata file] Q -->|derived tables
corrupted| R3[Rebuild from raw
with backfill: hours] Q -->|region unavailable| R4[Fail over to replica
region: per runbook] ``` Region failure is the expensive case. Replicating data, metadata and catalog to a second region, and keeping engines ready there, costs storage, transfer and standby compute. Many platforms accept a longer RTO for analytics than for operational systems, and replicate only the raw layer and the most critical tables. Residency rules apply to the replica too: a dataset whose residency rule requires the EU may only be replicated to another EU region. A plan that has not been tested is a guess. Restore exercises, such as recovering a dropped table in a test environment or rebuilding a gold table from raw, show whether runbooks work and how long they take. ## Key terms - Recovery point objective (RPO): the maximum acceptable data loss, as time. - Recovery time objective (RTO): the maximum acceptable time to restore. - Bucket versioning: keeping previous versions of overwritten or deleted objects. - Object lock: write-once retention that prevents deletion for a period. - Rollback: making an earlier snapshot the current state of a table. - Failover: switching service to a replica in another location. ## Example An engineer accidentally runs DROP TABLE shop.fct\_daily\_sales in Spark on a Monday morning. Without PURGE, Spark removes the table from the catalog and leaves its files in the bucket, which has versioning enabled. Some engines, such as Trino, and some catalogs delete the files as well, so check what yours does. 1. The on-call engineer finds the last metadata file in the table's location. 2. The table is registered again from it: ``` CALL lake.system.register_table( table => 'shop.fct_daily_sales', metadata_file => 's3://lake/shop/fct_daily_sales/metadata/00412-8c1e.metadata.json' ); ``` 1. Grants are reapplied from the governance files in version control. 2. Freshness checks confirm the table is current; dashboards work again after 25 minutes. Had the files been purged, they could have been restored from the versioned bucket by removing their delete markers, well within the table's RTO of four hours. Rebuilding fct\_daily\_sales from clean\_orders and clean\_order\_items, the fallback, takes several hours and could miss it. ## Pitfalls - Relying on storage durability. It protects against hardware loss, not against deletes and bad writes. - Backing up data but not the catalog. Tables then need to be registered again by hand, and permissions are lost. - Snapshot expiry shorter than detection time. A bad write found after ten days cannot be rolled back when snapshots are kept for seven. - Replicas that break residency rules. - Untested runbooks. The first real restore is the wrong time to find missing steps. ## Learn more - Amazon S3, using versioning: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html - Apache Iceberg, Spark procedures (rollback and register\_table): https://iceberg.apache.org/docs/latest/spark-procedures/ ## Related [Residency and regions]() · [Privacy lifecycle]() · [Observability and SLOs]() · [Backfill and replay]()