Cost and FinOps#
Optional
Understanding where a data platform's money goes, attributing it to the teams and products that cause it, and reducing it safely, as a shared routine of engineering and finance called FinOps.
Before reading
Why it matters#
Cloud data platforms are billed by use. A forgotten cluster or a table that is never cleaned up costs money every day, and one badly written query adds up fast: a dashboard tile that rescans the full orders table every five minutes reads hundreds of terabytes a month. The bill arrives a month later with no explanation. Cost that nobody can attribute is cost nobody can reduce.
How it works#
Costs come from a few places.
Compute is usually the largest share: query engine clusters, Spark jobs, warehouse credits, stream processors, the orchestrator. It is driven by how much data queries read, how long clusters run, and whether they sit idle.
Storage is cheaper per unit but grows without limit unless managed: raw data kept forever, old snapshots never expired, orphan files, duplicated tables, logs. Request charges on object storage, for many small files, add to it.
Data transfer is charged when data leaves a region or cloud: reading across regions, replicating for disaster recovery, sharing with recipients elsewhere, exporting to other systems.
Licences and managed services add fixed or per-use fees: catalogs, BI tools, reverse ETL, observability tools.
Attribution comes first. Resources are tagged with the owning team and product. Queries carry tags or run under identities that identify the team, dashboard or pipeline that sent them. Engines and warehouses record bytes scanned and time used per query; joined with the tags, they show cost per team, per data product and per dashboard. Showback reports this cost to teams; chargeback bills it to their budgets.
flowchart LR
B[Monthly bill:<br/>compute, storage,<br/>transfer, licences] --> A[Attribute by<br/>tags and query identity]
A --> T1[Per team]
A --> T2[Per data product]
A --> T3[Per dashboard<br/>or pipeline]
T1 & T2 & T3 --> R[Review top items,<br/>fix the largest first]Reducing cost safely follows the same order as any optimisation: measure, fix the biggest item, measure again. Common fixes, roughly from cheapest to hardest:
- Stop waste: idle clusters that never scale down, abandoned tables and pipelines, development environments left running.
- Read less: partition pruning, avoiding SELECT *, incremental models instead of full rebuilds.
- Store less: snapshot expiry, orphan removal, retention rules, compaction, colder storage classes for old data.
- Right-size compute: autoscaling, separating heavy batch from interactive work, committed-use discounts for steady load.
- Change architecture: fewer copies, fewer engines, or moving a workload to a cheaper engine.
Cost is a quality to watch like freshness, ideally as a unit cost, such as cost per million orders processed, so growth and waste can be told apart. Budgets per team, alerts on daily spend, and a weekly look at the most expensive queries stop a surprise from growing for a month.
Key terms#
- FinOps A practice of shared ownership of cloud cost between engineering and finance.
- Cost attribution Assigning spend to teams, products or workloads.
- Showback and chargeback Reporting cost to teams, or billing it to their budgets.
- Resource tagging Labelling cloud resources with owner and purpose.
- Egress Data transferred out of a region or cloud, usually charged.
- Unit cost Cost per unit of value, such as per dashboard view or per million orders processed.
Example#
The shop's compute bill doubles in a month. Query logs, which the platform collects from the engine into a table such as ops.query_history (in Trino, through an event listener), show the cause when grouped by query tag:
SELECT query_tag, count(*) AS runs,
sum(bytes_scanned) / 1e12 AS tb_scanned
FROM ops.query_history
WHERE started_at >= DATE '2026-10-01'
GROUP BY query_tag
ORDER BY tb_scanned DESC
LIMIT 5;
One dashboard tile, added three weeks earlier, refreshes every five minutes and filters with format_datetime(ordered_at, 'yyyy-MM-dd'), so it reads the whole orders table each time: about half of all bytes scanned (see Query performance on lake tables). Rewriting the filter as a range on ordered_at and refreshing hourly cuts its scans by more than 99%, and the bill returns to normal.
Pitfalls#
- No tags. Without attribution, every cost discussion is a guess.
- Cutting blindly. Shortening snapshot retention or shrinking clusters can break SLOs or time travel that someone relies on.
- Optimising small items. Measure first; the top few queries or tables usually dominate.
- Forgetting transfer. Cross-region reads and replication appear only on the bill.
- Cost reviewed once a year. Watch it weekly, with alerts on sudden changes.