Module 05 · Communication patterns · Lesson 9 of 10
Delivery semantics
At-most-once, at-least-once and the myth of exactly-once. Where duplicates and losses come from, why acknowledgement order decides the guarantee, and the outbox, inbox and idempotency patterns that give effectively-once end to end.
Why it matters
Every message system eventually produces a duplicate or loses a message, and which one it does is a choice you make, often without noticing, by the order of two lines of code. "Exactly-once" is the phrase everyone wants and no network can promise; what you can build is a system where duplicates are harmless and losses are impossible. This lesson is the difference between a design that sounds right and one that survives a worker crash at the worst moment.
The idea
Delivery semantics describe what a consumer observes when the network, the broker or the consumer fails mid-flight.
| At-most-once | At-least-once | Exactly-once (effectively) | |
|---|---|---|---|
| Guarantee | Delivered zero or one times; may be lost | Delivered one or more times; never lost, may duplicate | Each message's effect applied once |
| How | Acknowledge before processing (or fire and forget) | Acknowledge after processing | At-least-once plus deduplication or atomic processing and acknowledgement |
| Failure mode | Crash after ack, before processing: message gone | Crash after processing, before ack: redelivered | Needs state to detect repeats; that state must be durable and atomic with the effect |
| Fits | Metrics samples, logs, presence pings: losing one is fine | Almost everything with business meaning | Payments, inventory, anything where double application is a bug |
Where it is decided: the ack
At-least-once is the default you want for anything that matters, which means every consumer must tolerate duplicates. Everything below is about making that true.
Idempotent consumers
The cheapest route to effectively-once. Give every message a unique id. The consumer keeps a durable record of processed ids and skips repeats. The subtle requirement: the "record this id" write must be atomic with the effect, or you have simply moved the gap. Patterns, in order of preference:
- 1
Naturally idempotent operations
Set a status rather than toggle it; upsert by key rather than insert; store the final value rather than apply a delta. Repeats are then no-ops for free. - 2
Inbox table in the same transaction
Insert the message id into a processed-messages table with a unique constraint in the same database transaction as the business write. A duplicate fails the insert and the transaction rolls back untouched. - 3
Conditional writes
Carry a version or sequence number; apply only if the stored version is older. Stale or repeated messages are rejected by the database, not by application logic. - 4
External dedup cache
A set of recent ids in Redis with a TTL. Fast, but not atomic with the effect; acceptable for low-stakes side effects, not for money.
The dual-write problem and the outbox
A service that updates its database and then publishes an event does two writes to two systems. If it crashes between them, the database says "order placed" and no event exists, or (publishing first) an event exists for an order that never committed. There is no way to make two independent systems commit together without a coordinator. The transactional outbox fixes it: write the event into an outbox table in the same database transaction as the business change, and have a relay (a poller or change data capture) publish outbox rows to the broker, marking them sent. The relay is at-least-once, so events may publish twice, which brings you back to idempotent consumers, which you already have.
Broker-level exactly-once
Some brokers offer transactional producers and consumers: Kafka can commit a consumer's offset and its produced output messages atomically, so a stream processor that reads from one topic and writes to another applies each input once within the Kafka world. This is real and useful for stream processing pipelines. It does not extend to side effects outside the broker: sending an email, charging a card or writing to your own database still needs the patterns above.
Loss, not just duplication
At-least-once also demands that nothing is dropped. Producers must wait for the broker's durable acknowledgement (all replicas, not just the leader) before considering a message sent. Consumers must not auto-commit offsets before processing. Retries must be bounded and failures routed to a dead-letter queue rather than skipped, which the next lesson covers.
Trade-offs
At-most-once is the cheapest and only acceptable where loss is fine. At-least-once with idempotent consumers is the workhorse: a little dedup state per consumer buys correctness under every failure. End-to-end effectively-once with outbox and inbox tables costs a relay process, two extra tables and a habit of thinking about atomicity, which is a small price for never explaining a double charge.
In an interview
Say the phrase and then show the mechanics: "Delivery is at-least-once, so every consumer is idempotent. Order placement writes the order and an outbox row in one transaction; a relay publishes to Kafka. Each consumer inserts the event id into its inbox table in the same transaction as its own write, so a redelivery rolls back as a duplicate key. That gives effectively-once end to end without pretending the network can do it." If someone says "exactly-once", answer: "Within Kafka for stream processing, yes; across systems it is at-least-once plus deduplication."
Check yourself
A service updates its database and then publishes an event. It crashes after the database commit and before the publish. What happened, and what pattern prevents it?