Module 15 · Big data processing · Lesson 5 of 10
Data lakes
Cheap object storage holding raw and processed data of every shape, read by many engines. Schema on read, columnar formats and partitioning, the zone layout from raw to curated, the catalog that keeps it findable.
Why it matters
Clickstreams, logs, sensor data, images, model features and database exports add up to petabytes that are too varied and too voluminous for a warehouse's managed storage. A data lake stores all of it cheaply in object storage, in open formats any engine can read, and defers deciding how it will be used. That flexibility is the reason lakes underpin most large data platforms, and the lack of discipline it permits is the reason so many become swamps nobody trusts.
The idea
A data lake is a repository of data in object storage (S3, GCS, Azure Data Lake Storage) organised by directory-like key prefixes and stored in open file formats. Compute is separate: Spark, Trino, Athena, Flink and machine-learning frameworks read the files directly. Data is stored first and interpreted later (schema on read), in contrast to a warehouse that requires a schema on write.
Formats and layout
- 1
Columnar formats
Parquet and ORC store each column contiguously with compression and per-block statistics, so a query reading 3 of 200 columns reads 1.5 percent of the bytes and skips blocks whose min and max cannot match. Use them for anything queried; keep raw JSON or CSV only in the raw zone. - 2
Partitioning by key prefix
Lay data out as table/date=2026-11-08/region=eu/part-0001.parquet so engines prune entire directories from filters on date and region. Partition on columns that appear in most filters and have moderate cardinality; too many partitions means too many small files. - 3
File sizes
Hundreds of megabytes per file. Thousands of tiny files from streaming writes kill query performance and metadata operations; compaction jobs merge them. - 4
Zones
Raw (exactly as received, immutable, replayable), cleaned or standardised (deduplicated, typed, in Parquet), curated (modelled datasets for consumers). Each zone has owners and retention rules.
The catalog
Files without metadata are a swamp. A catalog (Hive Metastore, AWS Glue, Unity Catalog, Polaris) maps table names to locations, schemas and partitions so engines can query events instead of guessing paths, and records owners, descriptions, lineage and access policies. Without it, nobody knows which of three similar folders is current, what a column means, or who to ask when it breaks.
Why lakes become swamps
| Swamp symptom | Cause | Remedy |
|---|---|---|
| Nobody knows what data exists | No catalog or ownership | Catalog every dataset with an owner and description |
| Queries return inconsistent results | Concurrent writers, partial writes, no transactions on plain files | A table format with atomic commits (next lessons: lakehouse) |
| Schemas drift silently | Schema on read with no contracts | Schema enforcement in the cleaned zone; registry for event formats |
| Slow queries | Small files, JSON everywhere, no partitioning | Columnar formats, partitioning, compaction |
| Compliance risk | Personal data copied everywhere without tracking | Access control by zone and column, lineage, deletion workflows |
Lake versus warehouse
A lake is cheaper per terabyte, holds any data type, and serves many engines including machine learning, but plain files lack transactions, fine-grained updates and the performance and governance a warehouse provides out of the box. A warehouse gives fast, governed SQL over structured data at higher cost and with its own storage. Historically companies ran both and copied between them; the lakehouse lesson shows how table formats bring warehouse properties to lake storage.