Module 15 · Big data processing · Lesson 7 of 10
Data lakehouse
Warehouse guarantees on lake storage: open table formats (Iceberg, Delta Lake, Hudi) add atomic commits, schema evolution, time travel and row-level updates on top of Parquet in object storage. How the metadata layer works.
Why it matters
Running a lake for cheap, open, any-shape storage and a warehouse for fast, governed SQL means two copies of the data, two governance models and pipelines between them that drift. The lakehouse removes the split by giving files in object storage the properties that made warehouses trustworthy: atomic writes, consistent reads, updates and deletes, schema enforcement. It has become the default architecture for new data platforms, and the table format underneath it is the concept to understand.
The idea
A lakehouse stores data as open columnar files (Parquet) in object storage, plus a table format that adds a transactional metadata layer on top. The metadata records exactly which files make up the table at each version. A write creates new files and then atomically commits a new metadata version pointing to them; readers always see a complete version, never a half-written one. Any engine that understands the format (Spark, Trino, Flink, warehouse engines, Python libraries) can read and write the same tables.
What the table format adds
- 1
Atomic commits and isolation
Writes commit by atomically swapping the current metadata version (a compare-and-swap in the catalog or a conditional object write). Concurrent writers use optimistic concurrency: if the version moved, re-check for conflicts and retry. Readers get snapshot isolation. - 2
Row-level updates and deletes
MERGE, UPDATE and DELETE work on lake tables, needed for CDC ingestion, corrections and privacy deletion. Copy-on-write rewrites affected files at write time (fast reads); merge-on-read writes delete markers and new rows, merged at read time (fast writes); compaction reconciles. - 3
Schema evolution
Add, rename, reorder and widen columns safely, tracked by column id rather than position, so old files remain readable. - 4
Time travel
Query the table as of a version or timestamp, reproduce a model's training data, roll back a bad write by pointing to an earlier snapshot. - 5
Hidden partitioning and statistics
Iceberg derives partitions from column transforms (day of timestamp) so queries do not need to know the layout, and per-file statistics enable pruning without listing directories.
The three formats
| Apache Iceberg | Delta Lake | Apache Hudi | |
|---|---|---|---|
| Origin | Netflix | Databricks | Uber |
| Strengths | Engine-neutral, hidden partitioning, partition evolution, broad catalog support | Deep Spark and Databricks integration, mature streaming writes | Record-level upserts and incremental pulls, strong for CDC ingestion |
| Metadata | Snapshot tree of manifest files | Transaction log of JSON and checkpoint files | Timeline of commits with indexes |
| Trend | Widely adopted as the interoperable standard | Converging with Iceberg via shared readers | Common in upsert-heavy pipelines |
What it unlocks
One copy of data serving BI, data science and machine learning; streaming and batch writing to the same tables; CDC from operational databases applied as merges into lake tables; GDPR deletion in the lake; reproducible datasets via time travel; and freedom to switch query engines without migrating data. Many cloud warehouses now read and write Iceberg tables directly, which makes "lake versus warehouse" a choice of engine rather than of storage.
Operating one
Table maintenance is not optional: compact small files, expire old snapshots (time travel has a storage cost), clean orphaned files from failed writes, and rewrite manifests as tables grow. Choose copy-on-write for read-heavy tables and merge-on-read for frequently updated ones. Use one catalog per platform so every engine agrees on the current version.