Streaming engines

Why it matters

The batch-versus-stream lesson said streaming is hard because of state, time and failure. Streaming engines are the machinery that makes it tractable: they distribute a computation over partitions, keep its state fault-tolerant, reason about event time, and recover from crashes without losing or double-counting. When an interview asks for a real-time fraud detector, a live leaderboard or a continuously updated feature store, the credible answer names how the engine handles these three problems.

The idea

A streaming job is a dataflow graph of operators (source, map, filter, key by, window, aggregate, join, sink) running continuously. The engine parallelises each operator across tasks, routes records between them by key, and gives each keyed operator local state (counts, windows, last-seen values) that is partitioned by key just like the data.

Time: windows and watermarks

Engines window by event time. Tumbling windows are fixed and non-overlapping (per minute); sliding windows overlap (last 10 minutes, every minute); session windows close after a gap of inactivity per key. A watermark is a timestamp flowing with the data that asserts "no more events older than T are expected", typically the maximum event time seen minus an allowed delay. When the watermark passes a window's end, the engine emits the result. Events arriving after the watermark are late: dropped, sent to a side output, or used to update and re-emit within an allowed lateness. Choosing the watermark delay trades latency (emit sooner) against completeness (wait for stragglers).

Exactly-once with checkpoints

Flink periodically injects barriers into the stream from the sources; when an operator has received a barrier on all inputs it snapshots its state, and when every operator has done so the checkpoint, including source offsets, is complete. On failure the job restarts from the last checkpoint: state is restored and sources rewind to the recorded offsets, so every event affects state exactly once. Making output exactly-once additionally needs transactional or idempotent sinks: Kafka transactions committed with the checkpoint, or upserts by key into a database. Kafka Streams achieves the same within Kafka using transactions over its changelog topics.

State at scale

State can be gigabytes to terabytes per job. Engines keep it in memory or in an embedded store (RocksDB) on local disk, checkpoint incrementally to object storage, and redistribute it by key when the job is rescaled. Large state needs TTLs so keys for inactive users expire, otherwise state grows forever. Queries against state (the current count for user 42) can be served from the engine or, more commonly, from a sink table.

Backpressure

When a sink or operator is slower than its input, buffers fill and the engine slows upstream operators down to the slowest stage rather than dropping data or exhausting memory. Lag accumulates in the source log, which is safe because the log is durable. Consumer lag is the primary health metric; sustained growth means scale out or fix the slow stage.

Joins on streams

Stream-table joins enrich events with the latest value from a table (a user's tier), where the table is itself a changelog stream materialised as state. Stream-stream joins match events from two streams within a time window (an ad click with its impression within 30 minutes), holding both sides in state until the window closes. Temporal joins look up the table version valid at the event's time, which keeps replays deterministic.

Choosing an engine

Engine	Model	Strengths	Fits
Apache Flink	True streaming, per-event, distributed cluster	Low latency, rich event-time and state, large state, exactly-once, SQL and DataStream APIs	Complex stateful streaming at scale: fraud, CEP, real-time features, CDC pipelines
Kafka Streams	Library embedded in your service; Kafka as storage	No separate cluster; scales with service instances; exactly-once within Kafka	Kafka-centric microservices doing stateful transforms and aggregations
Spark Structured Streaming	Micro-batch by default (continuous mode limited)	Same API and engine as Spark batch; easy for teams already on Spark and lakehouse tables	Pipelines tolerant of seconds of latency; unified batch and streaming over Delta or Iceberg
Managed services	Hosted Flink, Dataflow (Beam), Kinesis Analytics	Less operations	Teams without a streaming platform team