Skip to content

Orchestration

Orchestration decides when each step of the platform's pipelines runs, in what order, and what happens when a step fails.

Before reading

dbt

Why it matters

A platform runs hundreds of jobs: loads, transformations, tests, exports, maintenance. Each depends on others finishing first. Run on independent timers, they start before their inputs are ready, fail silently or produce half-finished results. An orchestrator makes dependencies explicit, runs each step when its inputs are ready, retries failures, and shows in one place what ran, what failed and what is late.

How it works

The unit of work is a task: run a dbt model, load a file, call an API, compact a table. Tasks are arranged in a directed acyclic graph (DAG): an arrow from A to B means B runs only after A succeeds. The orchestrator's scheduler starts a run of the graph, launches each task when its upstream tasks have finished, and records the result.

Runs start in three ways. On a schedule, such as every night at 02:00. On an event, such as a file arriving in a bucket or a message on a topic. Or when an upstream dataset is updated. A sensor is a task that waits for a condition, such as "today's partition exists in raw_orders", before letting downstream tasks start.

Two styles of orchestration have emerged. Task-based orchestration describes what to run: a pipeline is a list of tasks with dependencies, and the orchestrator knows tasks, not data. Asset-based orchestration describes what should exist: each data asset, such as the table clean_orders, is declared with its upstream assets and the code that builds it, and the orchestrator works out what to run to bring assets up to date. Asset-based tools show the freshness and lineage of every table and run only what is stale. Apache Airflow, the most widely used orchestrator, is task-based at its core; it added data-aware scheduling in version 2.4 (as datasets), renamed and extended as assets in Airflow 3 (April 2025, checked October 2026). Dagster is built around assets. Prefect and Kestra are other common choices, and cloud providers offer managed Airflow.

flowchart LR
  S{{"Trigger: raw_orders_day_complete<br/>(asset event)"}} --> B["dbt build clean_orders,<br/>clean_order_items<br/>(models and tests)"]
  B -->|pass| F["dbt build fct_daily_sales+<br/>(models and tests)"]
  B -->|fail| A[Alert owner,<br/>stop downstream]

Failures are normal and must be handled by design. Retries with a delay recover from temporary problems, such as a busy engine or a network error; they are safe only if the task is idempotent. Timeouts stop tasks that hang. A failed task stops its downstream tasks, and the orchestrator alerts the owner. Each run is tied to a logical date or data interval, the period it processes, rather than to the clock, which is what makes reruns and backfills possible. In Airflow 3, runs triggered by assets have no logical date, and backfill works only for time-scheduled DAGs (checked October 2026).

The orchestrator should coordinate, not compute. Heavy work runs in the engine, dbt or a container the orchestrator launches; the orchestrator itself only tracks state. Business logic in orchestrator code, such as SQL strings inside task definitions, is hard to test and review.

Key terms

  • Task: one unit of work, such as running a model or loading a file.
  • DAG: a directed acyclic graph of tasks and their dependencies.
  • Scheduler: the component that starts runs and launches tasks when ready.
  • Sensor: a task that waits for a condition before downstream tasks start.
  • Asset: a data object, such as a table, that the orchestrator tracks and keeps up to date.
  • Logical date: the period a run processes, independent of when it runs.
  • Retry: re-running a failed task automatically after a delay.

Example

The nightly build should start when the previous day's orders are complete, not at a fixed time. raw_orders is fed continuously by CDC, so an asset event on every commit would start the build every five minutes. Instead, a completeness check confirms that the Kafka sink has committed offsets past midnight and then updates a separate asset, raw_orders_day_complete:

from airflow.sdk import Asset, dag, task
from datetime import datetime

raw_orders_day_complete = Asset("iceberg://lake/shop/raw_orders_day_complete")

@dag(schedule=[raw_orders_day_complete], start_date=datetime(2026, 1, 1), catchup=False)
def nightly_sales():
    @task.bash(retries=2)
    def build_clean():
        return "dbt build --select +clean_orders +clean_order_items"

    @task.bash(retries=2)
    def build_gold():
        return "dbt build --select fct_daily_sales+"

    build_clean() >> build_gold()

nightly_sales()

The completeness check task declares outlets=[raw_orders_day_complete], so Airflow records an asset event when it succeeds; a check running outside Airflow can post the event through Airflow's REST API instead. dbt build runs each model's tests right after building it. The DAG first brings clean_orders and clean_order_items up to date (a separate DAG also refreshes them hourly), and only if that passes builds fct_daily_sales and everything downstream (the trailing + selects downstream models). On a night the load is two hours late, the build starts two hours late instead of running on incomplete data.

Because this DAG is asset-triggered, history is reprocessed by running dbt directly for a date range, as in Backfill and replay, not through Airflow's backfill.

Pitfalls

  • Fixed times instead of dependencies. A job at 03:00 that assumes the 02:00 load has finished fails or runs on partial data when it is late.
  • Tasks that are not idempotent. Retries and reruns duplicate data.
  • Heavy processing inside the orchestrator. It becomes a bottleneck and hard to scale.
  • Logic hidden in DAG code. Keep transformations in dbt or the engine, where they are tested.
  • No owner on alerts. Failures that alert a shared channel are noticed late.

Learn more

  • Apache Airflow documentation, assets and data-aware scheduling: https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/asset-scheduling.html
  • Dagster documentation, assets: https://docs.dagster.io/guides/build/assets

Backfill and replay · Observability and SLOs · Environments and CI · Write-audit-publish