# Schema evolution Schema evolution is changing the structure of data over time, adding, removing, renaming or retyping fields, without breaking the systems that write and read it. > [!NOTE] > > **Before reading** > > [Schema-on-write and schema-on-read]() ยท [Kafka and Kafka Connect]() ## Why it matters Sources change constantly, and producers and consumers are upgraded at different times. A change that one side does not expect stops a pipeline, or worse, passes silently and fills a column with nulls. Knowing which changes are safe, and enforcing that automatically, turns most schema changes into non-events. ## How it works The basic question is whether data written with one version of a schema can be read with another. A backward compatible change lets readers using the new schema read data written with the old one. Adding an optional field with a default is backward compatible: old records simply get the default. A forward compatible change lets readers still on the old schema read data written with the new one. Adding an optional field is usually forward compatible too, because old readers ignore fields they do not know. A change that is both is fully compatible. Breaking changes are those that existing readers or writers cannot handle under the agreed compatibility mode, or that change meaning: renaming a field (seen as removing one field and adding another without a default), changing a type in a way that loses information (from text to number, or from decimal to integer), or changing what a field means while keeping its name. Removing a field is backward compatible, because new readers simply ignore it, but it is still a breaking change for every query that uses it. A change of meaning is the most dangerous, because no tool can detect it. In Kafka, a schema registry stores the versions of each topic's schema. Producers register the schema they write with; each record carries the schema's ID. Before accepting a new version, the registry checks it against the compatibility mode of the topic's subject (by default one subject each for key and value). In Confluent Schema Registry the default is BACKWARD, checked against the latest version only; BACKWARD\_TRANSITIVE checks against all versions. The mode also decides upgrade order: with backward compatibility, consumers are upgraded first; with forward compatibility, producers first; with full compatibility, either. ``` flowchart LR P[Producer with
schema v3] -->|register v3| R{Registry:
compatible with v2
under BACKWARD?} R -->|yes| OK[v3 accepted;
records carry its ID] R -->|no| NO[Rejected before
any data is written] OK --> C[Consumers read with
the writer's schema
and their own] ``` Table formats track schemas too. Iceberg identifies columns by ID, so adding, dropping and renaming columns, and widening types such as int to bigint, change only metadata and leave old files readable. Renaming in the table, however, does not help a consumer whose SQL uses the old name. Across the layers, changes are handled differently. The raw layer should accept them: new fields are added automatically, and the raw payload is kept, so nothing is lost while people decide what to do. The clean layer is deliberate: a new field is added to clean tables when someone decides what it means, and a breaking change is handled by mapping old and new versions to one clean column. Gold tables and published products change only through their contract, with notice to consumers. Renames are the classic problem. The safe sequence is expand and contract: add the new field, write both for a transition period, move readers to the new one, then remove the old one. ## Key terms - Backward compatibility New readers can read old data. - Forward compatibility Old readers can read new data. - Full compatibility Both directions. - Breaking change A change that existing readers or writers cannot handle under the agreed compatibility mode, or that changes meaning; removing a field consumers use is breaking even when the registry allows it. - Schema registry A service that stores schema versions and checks compatibility. - Compatibility mode The rule a registry enforces for a subject, usually one per topic key or value. - Expand and contract Making a breaking change in compatible steps. ## Example The orders service adds discount\_code. In the Avro schema it is optional with a default, so the registry accepts it under BACKWARD: ``` { "name": "discount_code", "type": ["null", "string"], "default": null } ``` raw\_orders gains the column automatically; old rows have null. clean\_orders adds it once its meaning is agreed: ``` ALTER TABLE shop.clean_orders ADD COLUMN discount_code varchar; ``` Later, the service wants to rename amount to total\_amount. Under BACKWARD, removing amount is allowed by the registry, but every query on amount would break. Instead the service writes both fields for a month, clean\_orders reads coalesce(total\_amount, amount), and amount is removed once nothing else uses it. ## Pitfalls - Compatibility mode NONE. Any change is accepted, and the first sign of trouble is a failed consumer. - Meaning changes without a schema change, such as amount switching from including tax to excluding it. Only contracts and communication catch these. - Auto-evolving clean or gold tables. Every raw change then reaches consumers without review. - Type narrowing. Converting a decimal to an integer silently loses information in old rows. - Renames in one step. Readers on the old name break immediately. ## Learn more - [Confluent documentation, schema evolution and compatibility]() - [Apache Iceberg, schema evolution]() ## Related [Data contracts]() [Apache Iceberg: table format basics]() [Change data capture]() [Testing data pipelines]()