Object storage and the S3 API¶

Object storage keeps data as whole objects, each a sequence of bytes plus a little metadata, addressed by a bucket and a key and read or written over HTTP.

Before reading

Data lake, warehouse and lakehouse

Why it matters¶

Object storage is where a lake or lakehouse keeps all of its data. It is cheap, very durable, effectively unlimited, and every engine can reach it. Its rules, no editing in place, no real folders, a price on every request, explain many design choices in file formats, table formats and file sizing.

How it works¶

A bucket is a named container with a region and an access policy. Inside it, every object has a key, its full name, such as shop/orders/data/00001.parquet. The slashes are just characters in the key. There are no folders: a console shows folders by grouping keys that share a prefix, and listing means asking for all keys that start with a prefix, returned in pages of up to 1,000 keys.

The operations are few. PUT writes a whole object. GET reads a whole object or a byte range of it. DELETE removes it, and LIST lists keys under a prefix. There is no append and no edit in place: changing one byte means writing the whole object again. There is no rename either. Moving an object is a copy followed by a delete, so renaming a "folder" of a million files is a million copies and is not atomic. Amazon S3 general purpose buckets work this way; S3 Express One Zone directory buckets added appends (2024) and atomic rename (2025), and Azure Data Lake Storage Gen2 renames atomically, but table formats do not rely on these. Large objects are uploaded in parts and appear only when the upload is completed.

Range reads matter for analytics. An engine reading a Parquet file first fetches the few kilobytes at the end of the file that describe its contents, then fetches only the byte ranges of the columns it needs.

flowchart LR
  E[Engine or tool] -->|PUT whole object| B
  E -->|GET object or byte range| B
  E -->|LIST keys under a prefix| B
  subgraph B[Bucket: lake]
    K["Objects, by key:<br/>shop/orders/data/00001.parquet<br/>shop/orders/data/00002.parquet<br/>shop/orders/metadata/v2.metadata.json"]
  end

Amazon S3 has offered strong read-after-write consistency since December 2020: once a write or delete succeeds, every following read and list sees it. Since 2024 it also supports conditional writes: write only if the key does not exist yet, or only if the object has not changed since it was read. That lets two writers coordinate on a single object. There are still no transactions across several objects, which is the gap that table formats such as Iceberg fill.

The S3 API began as Amazon's own interface and has become the common language of object storage. Self-hosted stores such as Ceph, SeaweedFS and Garage implement it (MinIO's open-source repository was archived in April 2026; checked October 2026), and so do several other clouds. Google Cloud Storage offers an S3-compatible interface next to its own; Azure Blob Storage uses its own API, which most engines also support. Compatibility is never complete. Consistency guarantees, conditional writes, versioning and lifecycle rules differ, so check the features your tools depend on.

You pay for storage per gigabyte-month, for each request (writes and lists cost more than reads), and for data transferred out of the region or cloud. Request rates are limited per prefix (for Amazon S3, at least 3,500 writes and 5,500 reads per second per prefix), and the store adds capacity by spreading load across prefixes.

Key terms¶

Bucket: a named container for objects, with a region and access policy.
Key: the full name of an object within a bucket.
Prefix: the start of a key, used for listing and access rules; what consoles show as a folder.
Range read: reading only part of an object by byte offsets.
Multipart upload: uploading a large object in parts that become visible together.
Conditional write: a write that succeeds only if the object is absent or unchanged.

Example¶

The shop's bucket, as keys (file names simplified). Raw exports land under raw/; Iceberg tables live under their table location.

s3://lake/
  raw/clickstream/2026/10/28/events-0001.json.gz
  raw/clickstream/2026/10/28/events-0002.json.gz
  shop/orders/metadata/v2.metadata.json
  shop/orders/metadata/snap-2.avro
  shop/orders/data/ordered_at_day=2026-10-28/00000-1-a41f.parquet
  shop/orders/data/ordered_at_day=2026-10-28/00001-1-a41f.parquet

Listing the prefix shop/orders/data/ordered_at_day=2026-10-28/ returns that day's files. Nothing in storage says they belong to a table; only the table's metadata says which files are current.

Pitfalls¶

Treating prefixes as folders. Renaming or moving a prefix copies every object under it.
Many small files. Each costs a request to write and to read, and listings get slow. Aim for files in the hundreds of megabytes.
Assuming every S3-compatible store behaves like Amazon S3. Test consistency, conditional writes and multipart behaviour before relying on them.
Lifecycle rules that delete or archive files a table still references. Let the table format remove its own files.

Learn more¶

Amazon S3 consistency model: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel
Amazon S3 conditional writes: https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html
Amazon S3 performance guidelines: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html

Columnar files and Parquet · Apache Iceberg: table format basics · Cost and FinOps · Residency and regions · Disaster recovery