Module 15 · Big data processing · Lesson 9 of 10
Kappa architecture
One streaming pipeline for everything: treat all data as a replayable log, and recompute history by replaying it through a new version of the job rather than running a separate batch path. Requirements (retention.
Why it matters
The Lambda architecture's flaw was writing everything twice. The Kappa architecture's answer is simple: do not have a batch layer. Keep every event in a durable log, process it with one streaming job, and when you need to recompute (a bug fix, a new metric, a changed definition), replay the log through a new version of that job. It is the design most event-centric platforms converge on, and it depends on a few properties that make it work or fail.
The idea
All data is an append-only, replayable log (Kafka, Pulsar, Kinesis with long retention, or a lakehouse table read as a stream). A single stream-processing job consumes it and maintains derived views in serving stores. There is no separate batch path; historical recomputation is done by the same code reading from an earlier offset.
The reprocessing playbook
- 1
Deploy the new job version alongside
Job v2 starts reading the log from the earliest retained offset (or from a snapshot point) and writes to a new output table or index, not the live one. - 2
Let it catch up
It processes history at full speed, usually much faster than real time, using more parallelism than steady state needs. - 3
Validate
Compare v2's output with v1's on overlapping periods; differences should be exactly the intended changes. - 4
Switch
Point readers (an alias, a view, a configuration flag) at v2's output once it is caught up to the head of the log. - 5
Retire
Stop v1 and delete its output. The log was never modified.
This is blue-green deployment applied to data pipelines, and it makes every change reversible.
What Kappa requires
| Requirement | Why | How it is met today |
|---|---|---|
| Long, cheap retention | History can only be recomputed if it is still in the log | Kafka tiered storage to object storage, compacted topics for state, or lakehouse tables as the long-term log |
| Correct stream processing | The single pipeline must be trusted to be right | Exactly-once state with checkpoints, event-time windows and watermarks (Flink, Kafka Streams) |
| Replay throughput | Reprocessing years of data must finish in hours, not weeks | Partitioned logs consumed in parallel; temporary scale-out for the catch-up job |
| Deterministic logic | Replaying must produce the same results as live processing | Event-time semantics, no wall-clock or external lookups whose results change over time (or versioned lookups) |
| Separate outputs per version | Live readers must not see a half-rebuilt view | Versioned tables and indexes behind an alias |
Limits
Replaying years of raw events can be slow and expensive compared with a batch engine over columnar files, especially for heavy joins; teams often replay from compacted or pre-aggregated intermediate topics or from lakehouse tables instead. Jobs that call external services during processing are hard to replay deterministically. Machine-learning training and very large ad-hoc analysis still belong to batch engines over the lake or warehouse; Kappa is about derived views and continuous metrics, not every analytical workload.
Kappa and the lakehouse
Lakehouse tables blur the boundary: a Flink job can write an Iceberg table continuously, and a batch engine can read the same table for history, so "replay" can be a batch read of the table with the same logic. The principle stays the same: one definition of the computation, one source of truth for the data, and recomputation by re-running rather than by maintaining a second path.