Module 15 · Big data processing · Lesson 8 of 10
Lambda architecture
Run a batch layer for complete, correct results and a speed layer for fresh approximate ones, and merge them at query time. Why it was invented, how the three layers fit, the cost of maintaining two pipelines with the same logic.
Why it matters
Early stream processors could not be trusted to produce correct results: they lost state on failure, double-counted on retry and could not handle late data. Batch jobs were correct but hours stale. The Lambda architecture made a pragmatic bet: run both, use the stream for freshness, and let the batch layer periodically overwrite the stream's approximations with correct numbers. Many large data platforms were built this way, and understanding why, and why most new designs avoid it, is a standard senior interview topic.
The idea
All incoming data is appended to an immutable master dataset. Three layers serve queries:
- 1
Batch layer
Periodically (hourly, daily) recomputes complete views from the entire master dataset with a batch engine. Results are correct and reproducible but stale by the batch interval plus run time. - 2
Speed layer
Processes new events as they arrive with a stream processor, producing incremental real-time views covering only the data since the last batch run. Results are fresh but may be approximate. - 3
Serving layer
Answers queries by merging the batch view with the speed layer's view for the recent window. When a batch run completes, the speed layer's view for that period is discarded.
Why it was attractive
The batch layer gives a correction mechanism: any bug in the speed layer, any lost or late event, any approximation is erased when the next batch run recomputes from the immutable master. Human errors are recoverable because raw data is never modified and views are always derived. The design accepted that streams were unreliable and contained the damage to a short window.
The cost: two pipelines
| Problem | Why it hurts |
|---|---|
| Logic implemented twice | The same aggregation is written once for the batch engine and once for the stream engine, in different APIs, by different people; they drift and produce different numbers |
| Merge complexity | The serving layer must combine views correctly at the boundary, handling overlaps and gaps |
| Operational burden | Two clusters, two deploy paths, two sets of monitoring and on-call knowledge |
| Debugging | A discrepancy could be in either path or in the merge |
| Cost | Data processed twice, compute for both always running |
The double implementation was the central complaint, and frameworks that ran one program in both modes (Summingbird, later Beam and Flink) only partly relieved it.
What changed
Stream processors became correct: exactly-once state with checkpoints, event-time windows and watermarks for late data, and replayable logs like Kafka with long retention. Once a stream can be replayed from the beginning to recompute results, the batch layer's correction role can be filled by reprocessing in the stream engine itself. That is the Kappa architecture, next lesson. Lakehouse tables also let streaming and batch write the same tables, softening the boundary.
Where Lambda still fits
When the historical computation is too large or complex for a streaming engine to replay economically (years of data, heavy joins, model training), when the fresh path deliberately uses a cheaper approximation (approximate counts from sketches, refined nightly), or when regulations require a batch-produced, reproducible figure as the system of record while operations want a live estimate. In those cases the dual path is a conscious trade, ideally with shared business logic in a library both engines use and a reconciliation check comparing them.