Change data capture

Why it matters

The system of record changes constantly, and a dozen other systems need to know: the search index, the cache, the warehouse, the recommendation features, the audit store. Having the application write to all of them is the dual-write problem multiplied. Polling the database for changes misses deletes and hammers it. Change data capture reads the stream of changes the database already writes for its own durability and replication, and publishes it. It is the quiet backbone of most modern data platforms and the cleanest way to keep derived systems in sync.

The idea

Every transactional database records committed changes in a log before applying them: Postgres's write-ahead log, MySQL's binlog, MongoDB's oplog. Replicas already consume that log. Change data capture attaches a reader to the same log, decodes each committed insert, update and delete into an event with before and after values, and publishes it, typically to Kafka, one topic per table, keyed by primary key.

Log-based versus query-based

The initial snapshot

A new consumer needs the existing data, not just future changes. Connectors take a consistent snapshot of the tables (reading at a log position) and then stream changes from exactly that position, so nothing is missed or duplicated between the two. Large tables snapshot in chunks, and incremental snapshotting lets new tables be added to a running pipeline without stopping it.

Ordering, delivery and schema

Events for one row are ordered (they come from one log and are keyed by primary key, so they land on one partition). Across rows and tables, ordering is the commit order of the log but consumers usually process partitions in parallel, so cross-entity ordering must not be assumed. Delivery is at-least-once after connector restarts, so consumers apply events idempotently (upsert by key). Schema changes are captured too: a schema registry versions the event format, and consumers must tolerate added columns. Large transactions arrive as many events; some connectors mark transaction boundaries for consumers that need them.

What it is used for

1. 1

   Search and cache sync

   Index updates and cache invalidations driven from committed changes, never from application code paths that might be skipped.
2. 2

   Feeding the warehouse and lake

   Replicate operational tables into analytics storage continuously instead of nightly dumps.
3. 3

   Outbox relay

   Publish the transactional outbox table without polling, the Transactions module's preferred relay.
4. 4

   Migrations and strangler figs

   Keep a new system's copy in sync with a legacy database during an incremental migration.
5. 5

   Audit and history

   Capture every change with before and after images for compliance.

Raw changes versus domain events

CDC on business tables publishes the schema: every column, internal names, and a consumer coupled to the producer's storage design. That is fine for replication into a warehouse or a search index owned by the same team. For integration across teams, publish domain events through an outbox table and CDC only that table, so consumers depend on a deliberate contract rather than on internal tables.

Trade-offs

CDC buys reliable, low-latency propagation of every committed change with minimal source load and no dual writes, and pays with connector infrastructure to run, logical replication configuration, schema coupling for raw-table capture, and at-least-once semantics that consumers must handle. It is the default way to keep derived stores in sync with a relational source of truth.

In an interview

Use it whenever a derived store appears: "Products live in Postgres. Debezium tails the WAL and publishes product changes to Kafka keyed by product id; an indexer upserts them into Elasticsearch and another consumer invalidates the cache. A new consumer gets a snapshot then the stream from the same position. Cross-team consumers read domain events from the outbox topic, not the raw table." That shows both the mechanism and the coupling judgement.