Idempotency

Why it matters

A client sends "charge the card". The server charges it and the response is lost to a timeout. The client has no way to know whether the charge happened, so it retries, and the customer is charged twice. Every distributed system has this gap between "request sent" and "response received", and the only robust fix is to make repeating a request harmless. Interviewers ask about idempotency because it separates people who have debugged a double charge from people who have not.

The idea

An operation is idempotent if performing it once or many times produces the same result. Setting a value is idempotent; adding to a value is not. Deleting a specific record is idempotent; "delete the newest record" is not. Idempotency is what makes retries safe, and retries are how distributed systems survive transient failures.

HTTP already tells you which methods are safe

Infrastructure relies on this: proxies and clients retry idempotent methods automatically and refuse to retry POST. If your PUT or DELETE has hidden side effects such as sending an email every time, you have broken a promise the whole stack depends on.

Idempotency keys make POST retry-safe

The client generates a unique key (a UUID) per logical operation and sends it in a header, Idempotency-Key. The server:

1. 1

   Looks up the key

   If seen before with the same request body, return the stored response without executing anything. If seen with a different body, return 422 or 409: the client reused a key for a different operation.
2. 2

   Reserves it atomically

   Insert the key with state 'in progress' before doing the work, with a unique constraint, so two concurrent retries cannot both execute. The loser waits or gets 409 and retries shortly.
3. 3

   Executes and stores the response

   Do the work, save the status code and body against the key, mark it complete. Ideally the work and the key update commit in one transaction.
4. 4

   Expires keys

   Keep them for as long as a client could plausibly retry, typically 24 hours, then delete.

Scope and design details

Scope keys to the authenticated caller so one tenant cannot replay another's. Hash the request body and store it so key reuse with different content is caught. Make the key store durable and shared across instances (database or Redis with persistence), because a retry may land on a different server. Keep the key check on the request path and the cleanup off it.

Idempotency without keys

Sometimes the operation can be made naturally idempotent instead. Use a client-supplied id for the resource (PUT /orders/{client-generated-id}), design state transitions that are no-ops when already applied (set shipped = true), or use conditional writes (If-Match, compare-and-set) so a stale retry is rejected rather than applied twice. Consumers of queues apply the same trick with message ids, which the Communication patterns module covers under delivery semantics.

Trade-offs

Idempotency keys add a storage lookup and write per request and a table that needs cleanup; they are worth it on anything with money or irreversible side effects, and overkill on cheap, naturally idempotent operations. Returning the stored response means the client sees the original outcome, including original errors, which is correct but occasionally surprising. Choosing the TTL trades storage against how long a client might legitimately retry.

In an interview

Volunteer it before being asked, whenever a design has a write with consequences: "Payments are POST with a client-generated Idempotency-Key. We reserve the key in the same database as the charge record, execute once, and replay the stored response on retry, so a network timeout cannot double-charge." Then show you know the subtle part: "The reservation is a unique insert, so two concurrent retries cannot both run."