Catalogs¶

A technical catalog is the service that maps table names to each table's current metadata, coordinates commits, and controls who may read or write each table.

Before reading

Apache Iceberg: table format basics

Why it matters¶

Every read and write of a lakehouse table goes through the catalog first. Engines that share a catalog see the same tables in the same state; engines with different catalogs do not. Because it sits on every path, the catalog is also the natural place to check permissions and hand out storage access. The business catalog, where people search for and document data, is a different tool, covered in Catalog and lineage.

How it works¶

A catalog has four jobs. It organises tables into namespaces, such as shop.orders. It stores, for each table, the location of the current metadata.json. It commits changes atomically: a writer says "the table is at snapshot 2, add snapshot 3", and the catalog moves the pointer only if the table is still at snapshot 2; otherwise the writer gets a conflict and must retry. And it decides who can do what.

Several kinds of catalog exist. The Hive Metastore is the older Hadoop-era service. A JDBC catalog keeps the pointers in a relational database. Cloud services, such as AWS Glue, run a catalog for you. The Iceberg REST catalog is an HTTP API defined by the Iceberg project. Any engine with a REST client can use any server that implements it, and the server can add features without engines having to change, so most engines and catalogs now support it.

The REST protocol covers configuration, namespaces, loading a table, committing changes, committing across several tables at once, and registering an existing table from its metadata file. Clients usually authenticate with an OAuth2 bearer token issued by the company's identity provider; services use the client credentials flow (see Authentication and authorization: OIDC and JWT). In short: the engine logs in to the company's central login service and receives a short-lived access token that it sends with each request. Some cloud endpoints use their own request signing instead, such as AWS SigV4.

sequenceDiagram
  participant E as Engine
  participant I as Identity provider
  participant C as Catalog (REST)
  participant S as Object storage
  E->>I: Get token (client credentials)
  I-->>E: Access token
  E->>C: Load shop.orders, with token, asking for vended credentials
  C->>C: Check the caller may read shop.orders
  C-->>E: Metadata location and short-lived storage credentials
  E->>S: Read metadata and data files with those credentials
  E->>C: Commit: expect snapshot 2, add snapshot 3
  C-->>E: OK, or 409 conflict

Credential vending closes a gap. If every engine holds permanent keys to the whole bucket, catalog permissions can be bypassed by reading files directly. With vending, the engine asks the catalog for access, the catalog checks its permissions and returns temporary storage credentials limited to that table's location. The alternative, remote signing, has the catalog sign each storage request instead. Either way the engine can reach only what the catalog allows.

Examples of implementations, as of October 2026: Apache Polaris, which became a top-level Apache project in February 2026; Lakekeeper; Apache Gravitino; Unity Catalog, open source and in Databricks; AWS Glue Data Catalog, which also offers an Iceberg REST endpoint; and Snowflake Open Catalog, a managed Polaris. Support for views, vending and multi-table commits differs between them.

The catalog holds little data, but it is critical. If it is lost, the data and metadata files still exist, but nothing records which metadata.json is current. Tables can be registered again from their latest metadata file, but choosing it is manual, and the permissions stored in the catalog are gone. Back it up like any production database.

Key terms¶

Technical catalog: maps table names to current metadata and coordinates commits.
Namespace: a grouping of tables, like a schema in a database.
Atomic commit: a pointer change that succeeds only if the table has not changed since the writer started.
Iceberg REST catalog: the HTTP API, defined by the Iceberg project, that engines use to talk to a catalog.
Credential vending: the catalog issuing short-lived storage credentials scoped to one table.
Remote signing: the catalog signing each storage request on the engine's behalf.

Example¶

Trino and Spark share one REST catalog, so both see the same shop.orders. In Trino, each data source the engine can query is configured as a "catalog", a word that clashes with the meaning on this page. This file configures one that points at the REST catalog:

connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=https://catalog.example.com/api/catalog
iceberg.rest-catalog.warehouse=lake
iceberg.rest-catalog.security=OAUTH2
iceberg.rest-catalog.oauth2.server-uri=https://idp.example.com/oauth2/token
iceberg.rest-catalog.oauth2.credential=trino:${ENV:TRINO_CATALOG_SECRET}
iceberg.rest-catalog.oauth2.scope=catalog
iceberg.rest-catalog.vended-credentials-enabled=true
fs.native-s3.enabled=true
s3.region=eu-west-1

Spark, in a notebook:

import os
from pyspark.sql import SparkSession

secret = os.environ["NOTEBOOK_CATALOG_SECRET"]
spark = (SparkSession.builder
    # plus the Iceberg Spark runtime and storage bundle jars (spark.jars.packages)
    .config("spark.sql.catalog.lake", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.lake.type", "rest")
    .config("spark.sql.catalog.lake.uri", "https://catalog.example.com/api/catalog")
    .config("spark.sql.catalog.lake.warehouse", "lake")
    .config("spark.sql.catalog.lake.oauth2-server-uri", "https://idp.example.com/oauth2/token")
    .config("spark.sql.catalog.lake.credential", f"notebook:{secret}")
    .config("spark.sql.catalog.lake.scope", "catalog")
    .config("spark.sql.catalog.lake.header.X-Iceberg-Access-Delegation",
            "vended-credentials")
    .getOrCreate())

spark.table("lake.shop.orders").count()

A row committed by Trino is visible to the next Spark query, because both read the same pointer.

Pitfalls¶

Two catalogs for one table. Each commits its own pointer, and writes through one are invisible to, or overwritten by, the other.
Permanent bucket keys on every engine. Catalog permissions then protect nothing.
No catalog backup or recovery test.
Assuming "REST compatible" means every feature. Check views, vending and multi-table commits for the engines you use.
Confusing the technical catalog with the business catalog. They solve different problems and are often different products.

Learn more¶

Iceberg REST catalog specification (OpenAPI): https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml
Trino, Iceberg REST catalog configuration: https://trino.io/docs/current/object-storage/metastores.html
Apache Polaris: https://polaris.apache.org/

Catalog and lineage · Authentication and authorization: OIDC and JWT · RBAC and ABAC · SQL query engines · Disaster recovery