Processing patterns: batch, micro-batch and streaming¶

Data can be processed in large scheduled chunks, in small frequent chunks, or continuously as each event arrives; the choice trades freshness against cost and complexity.

Why it matters¶

Every use case has a freshness need, and every step closer to real time costs more to build and run. Picking the pattern per use case, rather than one for the whole platform, avoids both stale reports and needlessly complex pipelines.

How it works¶

Batch processing collects data over a period, then processes it all in one run: every night, every hour. A run has a clear start and end, reads a bounded set of data, and can be repeated if it fails. It is the simplest pattern to build, test and fix. Freshness is limited by the schedule: a nightly batch means data up to a day old.

Micro-batch processing runs the same kind of job much more often, every few seconds to every few minutes, each time on whatever arrived since the last run. Each run is still a small batch with a start and an end, so the same tools and reasoning apply, but data arrives within minutes. Some streaming engines, such as Spark Structured Streaming, run streams as micro-batches internally, and many pipelines that look like streaming are micro-batches writing to tables.

Stream processing handles events continuously, one at a time or in tiny groups, as they arrive. It can react within seconds or less. But a stream has no end, so questions such as "how many orders today" need windows, state that survives restarts, and rules for events that arrive late. This makes streaming the most complex pattern to build and operate.

Real time is not a separate technique but a requirement: the result must be available within a stated delay. Seconds usually means streaming; a few minutes can be met with micro-batches.

flowchart TB
  subgraph Batch
    direction LR
    B1[Events collect<br/>all day] --> B2[Nightly run] --> B3[Report ready<br/>next morning]
  end
  subgraph Micro[Micro-batch]
    direction LR
    M1[Events] --> M2[Run every<br/>5 minutes] --> M3[Table up to date<br/>within minutes]
  end
  subgraph Streaming
    direction LR
    S1[Each event] --> S2[Processed<br/>on arrival] --> S3[Stock alert within<br/>seconds]
  end
  Batch ~~~ Micro ~~~ Streaming

A platform commonly uses all three. Events are captured as a stream, written to raw tables in micro-batches, and modelled in batches. A streaming path is added only for the few cases that need seconds.

Key terms¶

• Freshness: how old the data is when someone reads it.
• Latency: how long processing takes, from an event arriving to its result being available.
• Bounded data: a finite set with a known end, as processed by a batch.
• Unbounded data: a stream with no end.
• Window: a slice of a stream, such as the last five minutes, used to compute results.
• State: what a stream processor remembers between events, such as running totals.

Example¶

Three shop needs, three patterns.

Building the sales report as a stream would add windows, state and late-event handling to answer a question nobody asks more than once a day. The stock alert is different: a message that arrives an hour after stock ran out is useless, so it justifies the extra complexity.

Pitfalls¶

• Choosing streaming because it sounds modern. Ask how stale the result can be, then choose the simplest pattern that meets it.
• Ignoring late data. In batch, an event that arrives after its day's run is either dropped or counted on the wrong day unless past days are reprocessed. Streaming has the same problem and needs explicit rules for it.
• Very frequent micro-batches writing many small files, which slows every later read.
• One pattern for everything. Mixed patterns are normal; results agree when every pattern uses the same table and metric definitions.

Learn more¶

• Tyler Akidau, Streaming 101: https://www.oreilly.com/radar/the-world-beyond-batch-streaming-101/
• Martin Kleppmann, Designing Data-Intensive Applications (1st edition, 2017), chapters 10 and 11. A 2nd edition appeared in 2026; chapter numbers differ.

Related¶

Ingestion patterns · Change data capture · Kafka and Kafka Connect · Late and out-of-order data · Stream processing