# 03 Ingestion

How data gets from source systems into the platform's raw layer: loading from databases, APIs and files, streaming through Kafka and capturing database changes; the problems every pipeline meets, such as duplicates, late events and changing schemas; how to design the events applications publish; and, optionally, the outbox pattern, device data and stream processing.

> [!NOTE]
>
> **Before starting**
>
> [01 Foundations](<https://wiki.4tm0.net/data/section-foundations.html>) and [02 Storage](<https://wiki.4tm0.net/data/section-storage.html>), in particular [Processing patterns](<https://wiki.4tm0.net/data/processing-patterns.html>) and [Apache Iceberg: table format basics](<https://wiki.4tm0.net/data/iceberg-table-format-basics.html>).

## Pages, in reading order

1. [Ingestion patterns](<https://wiki.4tm0.net/data/ingestion-patterns.html>): Ingestion is getting data from source systems into the platform's raw layer, either by copying everything each time (a full load) or only what changed since the last run (an incremental load).
2. [Kafka and Kafka Connect](<https://wiki.4tm0.net/data/kafka-and-connect.html>): Apache Kafka is a distributed log that stores streams of events durably and lets many applications write and read them independently; Kafka Connect is its framework for moving data between Kafka and other systems without writing custom code.
3. [Change data capture](<https://wiki.4tm0.net/data/change-data-capture.html>): Change data capture (CDC) turns every insert, update and delete in a database into an event, usually by reading the database's own transaction log, so other systems can follow its changes.
4. [Delivery guarantees and idempotency](<https://wiki.4tm0.net/data/delivery-guarantees-idempotency.html>): A delivery guarantee says how often a message can arrive when things fail: at most once, at least once, or exactly once. Idempotency is the property that processing the same message twice has the same effect as processing it once.
5. [Late and out-of-order data](<https://wiki.4tm0.net/data/late-and-out-of-order-data.html>): Events often reach the platform later than they happened and in a different order; pipelines must decide which time they use and how long they wait for stragglers.
6. [Schema evolution](<https://wiki.4tm0.net/data/schema-evolution.html>): Schema evolution is changing the structure of data over time, adding, removing, renaming or retyping fields, without breaking the systems that write and read it.
7. [Event design](<https://wiki.4tm0.net/data/event-design.html>): How to design the events that applications publish, so that they are useful as data: what an event represents, how it is named, which fields every event carries, and how it changes over time.
8. [Writing to Iceberg from Kafka](<https://wiki.4tm0.net/data/kafka-to-iceberg.html>): Moving events from Kafka topics into Iceberg tables, usually with a sink connector that buffers records, writes data files and commits them to the table at a fixed interval.
9. [The outbox pattern](<https://wiki.4tm0.net/data/outbox-pattern.html>) (optional): An application publishes events reliably by writing them to an outbox table in the same database transaction as its business change; change data capture then delivers the outbox rows to an event stream.
10. [IoT and device data](<https://wiki.4tm0.net/data/iot-and-device-data.html>) (optional): Data from many small devices, such as sensors, scanners and vehicles, sent as frequent small messages over unreliable networks, usually through an MQTT broker before it reaches the platform.
11. [Stream processing](<https://wiki.4tm0.net/data/stream-processing.html>) (optional): Continuous computation over unbounded event streams, keeping state between events, so that results such as counts, joins and alerts are updated within seconds of each event.

## Previous and next section

[02 Storage](<https://wiki.4tm0.net/data/section-storage.html>) · [04 Modelling and transformation](<https://wiki.4tm0.net/data/section-modeling.html>)