Module 14 · Distributed transactions · Lesson 1 of 5
Why distributed transactions are hard
A transaction across two databases or two services has no shared log, no shared locks and no single commit point. Why atomicity breaks at a service boundary, the dual-write problem, the three ways out (avoid, coordinate, compensate).
Why it matters
Inside one database, "debit A and credit B" is one transaction and either both happen or neither. Split A and B across two services and that guarantee evaporates: there is no shared write-ahead log to make the pair atomic, no shared lock manager to keep others from seeing half the change, and no single moment that counts as the commit. Most microservice architectures discover this after they have drawn the service boundaries, which is the wrong time. This lesson frames the problem so the rest of the module's patterns make sense.
The idea
A local transaction gets atomicity from one log (all or nothing on recovery), isolation from one lock manager or MVCC (nobody sees intermediate state) and a single commit record that is the moment of truth. A distributed transaction spans resources that each have their own log, locks and commit, so those properties must be rebuilt by a protocol, approximated by a pattern, or given up on purpose.
The dual-write problem
Any time one operation writes to two systems, a database and a queue, two databases, a database and a search index, a crash between the writes leaves them inconsistent, and there is no retry that fixes it without risking a duplicate on the side that succeeded. It is the most common consistency bug in service architectures and it is structural, not a coding error. The outbox lesson gives the standard fix for the database-plus-message case.
What breaks and why
| Property | Local transaction | Across services |
|---|---|---|
| Atomicity | The log makes all-or-nothing automatic | Each service commits independently; a failure between commits is permanent unless something undoes the first |
| Isolation | Locks or MVCC hide intermediate state | Other requests see service A's commit before service B's; there is no cross-service snapshot |
| Consistency | Constraints span all the data | Invariants that span services (total stock equals sum of reservations) cannot be checked atomically |
| Durability | fsync and replication | Each service's own durability; the coordination state itself must also be durable |
| Failure handling | Rollback | A crashed coordinator, a lost message or a timed-out participant each need explicit handling |
Two-phase locking across services would hold locks in every service while the protocol runs, which multiplies latency and contention and couples the availability of every participant, which is why nobody does it at scale.
Three ways out
- 1
Avoid: draw the boundary so the transaction is local
The cheapest fix is to put data that must change together in the same database and service. If orders and order items always change together, they are one aggregate in one place. Many distributed transactions are a symptom of over-decomposition. - 2
Coordinate: a protocol that makes the commit atomic
Two-phase commit (next lessons) gets participants to agree before anyone commits. It gives real atomicity at the cost of blocking, latency, a coordinator that must be durable, and participants that all speak the protocol. It fits a small number of homogeneous resources under one administrative domain. - 3
Compensate: accept temporary inconsistency and repair it
Sagas run a sequence of local transactions and undo completed ones if a later one fails. The system is eventually consistent, intermediate states are visible and must be designed for (order pending), and every step needs an idempotent compensation. It fits workflows across independently deployed services and external providers.
Choosing
Ask: can the boundary move so this is local? If yes, do that. If the participants are a few databases you control and the operation is short, 2PC inside a transaction manager or a distributed SQL database that implements it over consensus is acceptable. If the participants are services, especially with external providers in the chain, use a saga with an outbox for reliable messaging and idempotent steps. In all cases, make intermediate states explicit in the data model so the rest of the system can reason about them rather than being surprised.
Trade-offs
Avoiding distributed transactions costs design discipline in service boundaries. Coordinating costs latency, availability coupling and blocking risk. Compensating costs visible intermediate states and the work of writing and testing compensations. Each is appropriate somewhere; the error is choosing by habit rather than by the shape of the workflow.
In an interview
Name the boundary decision before the pattern: "Orders and items are one aggregate in one database, so creating an order is a local transaction. Payment is another service, so order plus payment is a saga: order pending, charge with an idempotency key, confirm or cancel with a refund compensation, messages via an outbox. We do not use 2PC across services because it couples availability and blocks on coordinator failure." That sequence shows you considered all three ways out.
Check yourself
A service writes a new order to its database and then calls the inventory service to decrement stock. The inventory call times out. What is true about the system state, and which category of fix applies?