# Late and out-of-order data Events often reach the platform later than they happened and in a different order; pipelines must decide which time they use and how long they wait for stragglers. > [!NOTE] > > **Before reading** > > [Delivery guarantees and idempotency]() ## Why it matters A report for yesterday that is computed once, at midnight, misses every event of yesterday that arrives afterwards. Mobile apps that were offline, retries, batch exports and slow partitions all deliver late. Without a rule for late data, numbers change unpredictably or stay wrong. ## How it works Every event has two times. Event time is when it happened, set by the device or system that produced it. Processing time is when the pipeline handles it. Ingestion time, when it reached the platform, sits in between and is what `_ingested_at` records. The gap between event time and processing time is the event's lateness. Most events arrive within seconds; a few arrive hours or days later. Out of order means events arrive in a different order from their event times. Kafka keeps order within a partition, but events from different devices, partitions or retries interleave. A pipeline that assumes events arrive sorted will mis-order them. Results that answer questions about the real world, such as sales per day or sessions per user, must be grouped by event time. Grouping by processing time is easier and always complete, but it puts a late event into the wrong day. ``` flowchart LR E1["Click at 23:58 on 28 Oct
(phone offline)"] -->|arrives 02:51
on 29 Oct| P[Pipeline] P -->|by event time| D1[Counted in 28 Oct] P -->|by processing time| D2[Counted in 29 Oct] ``` Batch pipelines handle lateness with a reprocessing window. Instead of computing only yesterday, each run recomputes the last few days of results by event time, picking up events that arrived since the last run. The window length is a trade-off between cost and completeness, and is chosen by measuring how late events actually arrive, for example the lateness that 99.9% of events stay under. Events later than the window are either dropped, counted in a separate late bucket, or picked up by a periodic larger backfill. Stream processors face the same question continuously. They track a watermark: an estimate, derived from the event times seen so far, that no events older than a certain time are still to come. When the watermark passes the end of a window, the window's result is emitted. Allowed lateness keeps a window open for updates a while longer, and events later than that are dropped, or sent to a side output if one is configured. A watermark is a heuristic, so the trade-off remains: wait longer for completeness, or emit sooner with some events missed. Results that can change after they were first published must be presented as such. A daily figure can be labelled provisional until its reprocessing window has passed, or the consumers can be told which days may still change. ## Key terms - Event time When the event happened. - Processing time When the pipeline handles the event. - Ingestion time When the event reached the platform. - Lateness The delay between event time and processing time. - Reprocessing window The number of recent days a batch job recomputes on each run. - Watermark (streaming) An estimate that no events older than a given time are still to come. - Allowed lateness How long a window still accepts updates after its watermark has passed. ## Example A customer clicks at 23:58 on 28 October with no signal; the phone sends the click at 02:51 on 29 October. The nightly session job ran at 01:00 and counted 28 October without it. The job recomputes the last three days of sessions on every run: ``` DELETE FROM shop.fct_daily_sessions WHERE session_date >= current_date - INTERVAL '3' DAY; INSERT INTO shop.fct_daily_sessions SELECT CAST(event_at AS date) AS session_date, count(DISTINCT session_id) AS sessions FROM shop.clean_clickstream WHERE event_at >= current_date - INTERVAL '3' DAY GROUP BY 1; ``` On 30 October the job picks up the late click and 28 October gains one session. Measured over a month, 99.95% of clicks arrive within two days, so three days covers nearly all of them; the rest are caught by a monthly backfill. Each statement is its own commit, so readers can briefly see the gap between them. A single MERGE, or a partition overwrite in engines that support it, avoids this. ## Pitfalls - Grouping by processing time for business results. Late events land on the wrong day. - Trusting device clocks blindly. Phones with wrong clocks send events from the future or the distant past; reject or clamp impossible times. - A reprocessing window chosen by guess. Measure the lateness distribution and revisit it. - Partitioning raw data by event time. A late event then lands in an old partition, adding small files there and forcing jobs that process by partition to revisit it; partition raw data by ingestion time and clean data by event time. - Consumers who think published numbers are final. Say which days can still change. ## Learn more - [Tyler Akidau, Streaming 102]() - [Apache Flink documentation, timely stream processing]() ## Related [Stream processing]() [Backfill and replay]() [Partitioning]() [Processing patterns: batch, micro-batch and streaming]()