Sagas

Why it matters

Order, pay, reserve, ship: four services, four databases, one business outcome. Two-phase commit across them would hold locks across network calls and block on coordinator failure, and the payment provider cannot prepare anyway. The saga is how real systems make multi-service workflows reliable: each step commits locally, and if something fails later the earlier steps are undone by explicit compensations. It is the pattern interviewers expect for any cross-service business flow, and the details (compensations, idempotency, intermediate states) are where the real design lives.

The idea

A saga is a sequence of local transactions T1, T2, ... Tn, each in one service, with compensating transactions C1, C2, ... that semantically undo their counterparts. If Tk fails, the saga runs Ck-1 down to C1 in reverse order. The whole flow is eventually consistent: between steps, the system is in an intermediate state that other parts of the system can observe and must be designed to tolerate.

Two ways to coordinate

For anything beyond three steps, orchestration is usually the better engineering choice because the flow is inspectable and changeable in one place. The orchestrator must itself be durable: its state in a replicated store, its progress resumable after a crash.

Designing the steps

1. 1

   Make every step idempotent

   Steps and compensations will be retried after timeouts. Key them on the saga id so repeats are no-ops: reserving stock twice for saga 9001 reserves once.
2. 2

   Design compensations as business actions

   A compensation is not a rollback; it is a new forward action with business meaning: refund, release, cancel, send an apology. It must be possible even after other things have happened, which is why some actions cannot be compensated (an email cannot be unsent) and must be moved after the pivot.
3. 3

   Order steps: compensatable, pivot, retryable

   Put steps that can be undone first; then the pivot step, which once done means the saga will complete (the charge, typically); then steps that cannot fail permanently and are simply retried until they succeed (send confirmation, update analytics). Nothing after the pivot needs a compensation.
4. 4

   Keep a saga log

   Record each step's start and outcome durably before moving on, so a crashed orchestrator resumes correctly and never re-runs a completed pivot.
5. 5

   Use semantic locks for intermediate state

   Mark the order as pending so other flows (a second checkout, a cancellation) see that a saga owns it and either wait or reject, since there is no isolation across services.
6. 6

   Bound the saga

   Give reservations expiries and sagas deadlines, so a stalled saga releases resources and lands in a known failed state instead of holding stock forever.

Messaging and the outbox

Each step's local transaction and the message that triggers the next step must not be split by a crash. The transactional outbox, next lesson, writes the message in the same database transaction as the step's state change and relays it afterwards. Combined with idempotent consumers, this gives sagas reliable at-least-once messaging without 2PC.

When things go wrong anyway

Compensations can fail too. Retry them with backoff; if they keep failing (the refund API is down for hours), park the saga in a manual-intervention state with a dead-letter record and alert a human. Never let a saga end silently half-done. Expose saga state on a dashboard keyed by the saga id, carry that id as the correlation id on every message, and alert on sagas stuck in a state beyond a threshold. Sagas replace database isolation with visibility, so observability is part of correctness.