Apache Iceberg on your own hardware: what the table format actually solves

Putting Parquet files in object storage and pointing a query engine at them does not give you a warehouse. Everything databases take for granted is missing: the certainty that a query sees a consistent state, that an aborted write leaves no half results, and that renaming a column does not break every report.

That is the gap Apache Iceberg closes. Not with a new file format — Parquet or ORC still sits underneath — but with a metadata layer that records which files belonged to a table at which point in time.

Snapshots instead of directories

The central idea is the snapshot. Every write produces a new metadata file containing the complete list of data files that belong to the table. A query reads that list first and the files second — never the directory.

Almost everything else follows. A write becomes visible when the pointer moves to the new snapshot, and not before: that is atomicity. Old snapshots stick around for a while: that is time travel. And because the file list is explicit, nothing has to list a directory — with millions of files in object storage that is the difference between seconds and minutes.

-- Today's state
SELECT sum(amount) FROM lakehouse.sales.orders;

-- State at month end, no copy, no backup
SELECT sum(amount) FROM lakehouse.sales.orders
  FOR TIMESTAMP AS OF '2026-06-30 23:59:59';

-- What changed since then?
SELECT count(*) FROM lakehouse.sales.orders
  FOR TIMESTAMP AS OF '2026-06-30 23:59:59';

For regulatory reporting and audit the second case matters most. The question “what did the figure look like on the reporting date” is otherwise answered with monthly copies that cost storage and drift apart.

Schema changes without rewriting

Iceberg identifies columns by stable IDs, not by name or position. Renaming is therefore a metadata operation, not a data operation. The same holds for adding, dropping and reordering columns, and for changing the partitioning.

In Iceberg, partitioning is a property of the table, not of the file path. Changing it does not require rewriting old data.

That sounds technical and in practice determines whether a data model can be revised. Where a partition change means a multi-day rewrite, it does not happen — and the table stays wrongly cut forever.

The catalog is the real dependency

Here it gets uncomfortable. Iceberg itself is open and specified. But something has to know which metadata file is a table's current snapshot, and that pointer swap has to be atomic. That is the catalog's job.

And catalogs are where vendor lock-in comes back. Anyone using a cloud provider's proprietary catalog has open tables in a closed index. The data is readable, but only a service you cannot take with you knows which version is authoritative.

So anyone whose goal is independence should not check whether Iceberg is supported, but which catalog is used and whether it can be operated in-house. That is the question almost always missing from tenders.

What Iceberg does not solve

• Speed. Iceberg speeds up metadata resolution, not the query. That is the engine's job through caching, vectorization and statistics.
• Small files. Frequent streaming writes produce many small files; compaction has to be planned and does not happen by itself.
• Permissions. Iceberg knows nothing about users. Row and column rights belong in the engine or the semantic layer.
• Old snapshots. Time travel costs storage. Without an expiry policy a table that never grows still grows.

That is not criticism but scope. Iceberg solves the bookkeeping. Everything else remains work — just no longer the work of turning files into a table.