Module 13 · Distributed fundamentals · Lesson 15 of 18
Distributed locks
Mutual exclusion across machines: a lock in Redis or a coordination service, with a TTL so a dead holder cannot block forever. Why a TTL alone is unsafe, the fencing token that fixes it, the Redlock debate.
The video walkthrough is part of Pro.
See what Pro includesWhy it matters
"Only one worker should process this order at a time" is one of the most common requirements in a distributed system and one of the most commonly mis-implemented. A lock that lives in a single Redis key with a timeout feels right and is unsafe in exactly the failure cases locks exist for. This lesson gives the correct shape, the honest limits, and the alternatives that are often simpler and safer than a lock.
The idea
A distributed lock grants one holder exclusive access to a resource across machines. Because the holder can crash while holding it, every distributed lock has an expiry (a lease or TTL); because the holder can be slow rather than dead, the lease can expire while the holder still thinks it has the lock. Everything about distributed locking follows from that second sentence.
The simple lock and its flaw
SET lock:order:42 <random-token> NX PX 30000 in Redis acquires a lock if the key does not exist, with a 30-second TTL and a token identifying the holder. Release is a script that deletes the key only if the token matches. This prevents one client from releasing another's lock. It does not prevent the failure below.
Fencing tokens make it safe
The lock service hands out a monotonically increasing fencing token with each acquisition (etcd's revision, ZooKeeper's zxid or sequential node number, a counter in a consistent store). The holder passes the token with every write to the protected resource, and the resource rejects any token lower than the highest it has accepted. The paused client's write with token 33 arrives after token 34 has been seen and is refused. This moves the enforcement from the lock holder, who cannot know it was paused, to the resource, which can. A lock without fencing is a lock that works except when it matters.
Where to host the lock
| Single Redis (SET NX PX) | Redlock (N independent Redis nodes) | Coordination service (etcd, ZooKeeper, Consul) | Database row | |
|---|---|---|---|---|
| Guarantee | Efficiency: avoids duplicate work; not correctness | Majority of N nodes; contested safety under clock jumps and pauses | Consensus-backed; leases plus revisions as fencing tokens | Transactional; a conditional update or SELECT FOR UPDATE |
| Fencing token | Not natively; add a counter with care | No | Yes | Yes (a version column) |
| Latency | Sub-millisecond | Several round trips | Low milliseconds | Low milliseconds |
| Fits | Best-effort deduplication, rate limiting, cache stampede guards | Rarely the right trade; prefer a coordination service | Correctness-critical exclusion | Locks scoped to data already in the database |
The Redlock debate: acquiring a lock on a majority of independent Redis nodes improves availability but still relies on timing assumptions (bounded pauses and clock drift) and does not provide fencing. For efficiency locks a single Redis is simpler and fine; for correctness locks use a consensus-backed service with fencing. Know which kind of lock you need before choosing.
Lease length and renewal
Set the lease long enough to cover normal work with margin, and renew it periodically from the holder while work continues (a watchdog), stopping work if a renewal fails. A short lease recovers quickly from crashes but needs frequent renewal and is more likely to expire during a pause; a long lease blocks the resource longer after a crash. Measure the lease on the monotonic clock. None of this replaces fencing; it reduces how often fencing has to save you.
Often you do not need a lock
- 1
Conditional writes
UPDATE orders SET status='shipped', version=version+1 WHERE id=42 AND version=7. If zero rows changed, someone else got there first. The database's atomicity is the lock, scoped exactly to the data. - 2
Idempotency
If processing twice is harmless (same result, same side effects deduplicated by key), let two workers race and discard the loser's outcome. - 3
Partition the work
Assign each resource to exactly one worker by hashing or consumer-group assignment, so there is nothing to contend over. - 4
Single-writer queues
Route all operations on a resource through one queue partition consumed by one consumer at a time.
Each of these is simpler than a lock and immune to the lease expiry problem, which is why experienced engineers reach for them first.