Module 12 · Microservices patterns · Lesson 5 of 8
Circuit breaker
Stop calling a dependency that keeps failing, fail fast instead, and probe for recovery. The closed, open and half-open states, choosing thresholds and windows, what to do when the circuit is open (fallbacks).
The video walkthrough is part of Pro.
See what Pro includesWhy it matters
When a dependency is down, the worst thing a caller can do is keep calling it: every call waits for a timeout, holds a thread and a connection, and adds load to a service trying to recover. Multiply by every instance of every caller and a single failing dependency drags the whole system into slow failure. The circuit breaker is the pattern that notices a dependency is failing, stops calling it, and gives it room to recover while callers fail fast and degrade. It is the most-asked resilience pattern in interviews because it is the one that stops cascades.
The idea
A circuit breaker wraps calls to a dependency and tracks their outcomes. It has three states.
- 1
Closed (normal)
Calls pass through. The breaker records successes and failures (errors, timeouts, slow responses) over a sliding window. - 2
Open (tripped)
When failures exceed a threshold (say 50 percent of at least 20 calls in 10 seconds), the breaker opens: calls fail immediately without reaching the dependency, returning an error or a fallback. The dependency gets no load from this caller. - 3
Half-open (probing)
After a cooldown (say 30 seconds), the breaker lets a small number of trial calls through. If they succeed, it closes; if they fail, it opens again and the cooldown restarts, often with backoff.
Choosing the parameters
Count failures as a rate over a minimum volume, not an absolute number, so one failure in a quiet period does not trip it and a flood of failures under load does. Treat slow calls as failures (a dependency responding in 5 seconds is as harmful as one returning errors), with a slow-call threshold near the timeout. Set the cooldown long enough for a dependency to recover from a restart or a deploy, typically 10 to 60 seconds, with exponential growth on repeated reopenings. Allow a handful of half-open trials so one unlucky probe does not keep the circuit open. Tune per dependency: a breaker on a cache can be aggressive; one on a payment provider should be conservative because its fallback is expensive.
What to do when it is open
A breaker without a fallback just converts slow failures into fast ones, which is already valuable (threads and connections are freed, users get a quick answer). A breaker with a fallback turns failure into degradation: serve cached or default data, hide the recommendations strip, queue the write for later, show "temporarily unavailable" for that feature only. The fallback is designed per call site and tested, because an untested fallback is a second outage waiting inside the first.
Scope
| Scope | Behaviour | Use |
|---|---|---|
| Per dependency | One breaker for all calls to the inventory service | Default; matches the failure domain of a whole service outage |
| Per endpoint or operation | Separate breakers for inventory read and inventory reserve | When one operation is slow or broken while others are fine |
| Per instance (outlier detection) | Eject a single failing instance from the load-balancing pool | Sidecars and balancers; complements, does not replace, the per-dependency breaker |
| Global across callers | Shared state so all instances of a caller trip together | Rarely worth the coordination; per-instance breakers converge quickly on their own |
Where breakers live
In application libraries (Resilience4j, Polly, go-resilience), where they can call a fallback with full business context. In sidecars and meshes (Envoy outlier detection and circuit breaking), where they apply uniformly to all services without code but can only fail fast, not degrade. In API gateways, protecting upstreams from external traffic. Mature systems use the mesh for uniform fast-fail and library breakers where a business fallback exists.
Observability
A breaker's state is a first-class signal: alert when one opens, dashboard the open and half-open transitions, and record the fallback rate. An open breaker is often the earliest and clearest indicator of an incident in a dependency, visible before that dependency's own alerts fire.