03 Ingestion#

1 Ingestion patterns

Ingestion is getting data from source systems into the platform's raw layer, either by copying everything each time (a full load) or only what changed since the last run (an incremental load).

2 Kafka and Kafka Connect

Apache Kafka is a distributed log that stores streams of events durably and lets many applications write and read them independently; Kafka Connect is its framework for moving data between Kafka and other systems without writing custom code.

3 Change data capture

Change data capture (CDC) turns every insert, update and delete in a database into an event, usually by reading the database's own transaction log, so other systems can follow its changes.

4 Delivery guarantees and idempotency

A delivery guarantee says how often a message can arrive when things fail: at most once, at least once, or exactly once. Idempotency is the property that processing the same message twice has the same effect as processing it once.

5 Late and out-of-order data

Events often reach the platform later than they happened and in a different order; pipelines must decide which time they use and how long they wait for stragglers.

6 Schema evolution

Schema evolution is changing the structure of data over time, adding, removing, renaming or retyping fields, without breaking the systems that write and read it.

7 Event design

How to design the events that applications publish, so that they are useful as data: what an event represents, how it is named, which fields every event carries, and how it changes over time.

8 Writing to Iceberg from Kafka

Moving events from Kafka topics into Iceberg tables, usually with a sink connector that buffers records, writes data files and commits them to the table at a fixed interval.

9 The outbox pattern Optional

An application publishes events reliably by writing them to an outbox table in the same database transaction as its business change; change data capture then delivers the outbox rows to an event stream.

10 IoT and device data Optional

Data from many small devices, such as sensors, scanners and vehicles, sent as frequent small messages over unreliable networks, usually through an MQTT broker before it reaches the platform.

11 Stream processing Optional

Continuous computation over unbounded event streams, keeping state between events, so that results such as counts, joins and alerts are updated within seconds of each event.