Transactional outbox

Why it matters

Every service that changes its database and tells others about it faces the same gap: commit then publish loses the message on a crash; publish then commit announces something that may roll back. The outbox pattern closes that gap with nothing more exotic than a table and a background relay, and it is the mechanism under reliable sagas, event-driven architectures and change propagation to search indexes and caches. It is small enough to describe in a minute and important enough that interviewers listen for it.

The idea

Instead of publishing to the broker directly, the service inserts the message into an outbox table in the same database transaction as its business change. Either both commit or neither does, because they are one transaction. A separate relay reads new outbox rows, publishes them to the broker, and marks them sent. The business transaction never depends on the broker being available, and no message can exist for a change that did not commit.

Two kinds of relay

	Polling publisher	Change data capture (log tailing)
How	A worker periodically selects unsent rows (FOR UPDATE SKIP LOCKED), publishes, marks sent	A connector (Debezium) reads the database's replication log, sees outbox inserts, publishes them
Latency	Poll interval, typically 100 ms to a few seconds	Near real time
Database load	Repeated queries on the outbox table	Negligible; reads the log the database already writes
Complexity	Simple; lives in the service	Another component to run and monitor; needs logical replication enabled
Ordering	Per worker; use one worker per aggregate shard or order by id	Log order, which is commit order
Fits	Small services, getting started	Many services, high volume, low latency

Both relays are at-least-once: a crash after publishing but before marking sent republishes the row. That is acceptable because consumers are made idempotent, and it is far better than the at-most-once of a direct publish that might never happen.

Consumers: the inbox pattern

The receiving service records each processed message id in an inbox table in the same transaction as its own state change. A redelivered message hits a unique constraint and is skipped. This makes the whole chain effectively-once end to end: outbox for the producer side, inbox for the consumer side, at-least-once in between. Where the consumer's effect is naturally idempotent (set status to shipped), the inbox can be skipped.

Ordering

Events for the same aggregate (one order) should arrive in order. Publish with the aggregate id as the partition key so they land on one partition, and have the relay publish rows in id order within an aggregate. Events for different aggregates may interleave freely. If strict global order were needed, there would be one partition and one relay, which is a throughput ceiling to avoid.

Keeping the outbox healthy

Index on (sent_at, id) so the poller's query is cheap. Delete or archive sent rows promptly (a nightly job or a short retention) so the table stays small. Alert on the age of the oldest unsent row: a growing backlog means the relay is down or the broker is unreachable, which is exactly the failure the pattern is designed to survive but not to hide. Keep payloads self-contained (the event carries what consumers need) or thin (ids only) consistently, as the pub/sub lesson discussed.

Variants and relatives

Listen to yourself: the service consumes its own published events to update its read models, keeping even internal projections consistent through the same path. Event sourcing goes further and makes the event log the source of truth, so there is no separate outbox; the Architectural patterns module covers it. Transaction log tailing without an outbox table (publishing raw row changes) works for data replication but leaks the schema; the outbox lets you publish well-shaped domain events instead.