Bulkheads

Why it matters

A ship's hull is divided into watertight compartments so a breach floods one section, not the vessel. Services need the same: when the recommendations dependency hangs, requests to it pile up and occupy every thread and connection in the service, and suddenly checkout, which never calls recommendations, is failing too. Bulkheads confine the damage. They are less famous than circuit breakers and arguably more fundamental, because they limit how much any single failure can consume before anything else notices.

The idea

A bulkhead is a bounded, dedicated pool of some resource (threads, connections, memory, instances, queue capacity) assigned to one dependency, one tenant or one class of work, so that exhaustion of that pool affects only that compartment. The rest of the system keeps its own pools and keeps working.

Forms of bulkhead

Form	Isolates	Mechanism
Per-dependency thread or connection pool	One downstream's slowness from others	Separate pools or semaphores per client; the classic Hystrix model
Concurrency limit per operation	Expensive operations from cheap ones	At most N in-flight exports; excess queued briefly or rejected
Separate instance pools	Traffic classes from each other	Dedicated deployments or node pools for interactive versus batch, or for a critical API versus everything else
Tenant isolation	Noisy tenants from quiet ones	Per-tenant quotas, partitions, queues, or dedicated shards for the largest tenants
Queue partitioning	One slow consumer or message type from the rest	Separate queues or partitions per type or tenant with their own consumers
Process or container isolation	Memory leaks and CPU spikes	Separate processes or pods for risky components, with resource limits

Sizing compartments

Each pool is a cap on how much of the system that compartment can consume. Size it from the dependency's expected concurrency (rate times latency, as in the connection pooling lesson) with headroom, and make the total of all pools no more than the service can actually run. A too-small pool rejects legitimate work; a too-large one defeats the purpose. Adaptive limits (TCP-style additive increase and multiplicative decrease on observed latency) can tune concurrency limits automatically and are built into some mesh and library implementations.

Bulkheads and circuit breakers together

They answer different questions. The bulkhead bounds how much a failing dependency can consume at any moment: at most 20 threads, ever. The circuit breaker decides whether to call it at all once it is clearly failing. Without a bulkhead, a breaker takes time to trip and in that window the dependency can exhaust the shared pool; without a breaker, a bulkhead keeps wasting its 20 threads on timeouts indefinitely. Together, the bulkhead caps the blast radius immediately and the breaker then stops the bleeding and probes for recovery.

Noisy neighbours in multi-tenant systems

One tenant running a bulk import can starve every other tenant's requests. Bulkheads here are per-tenant: fair-share schedulers, per-tenant concurrency limits and rate limits, separate queues so one tenant's backlog is not everyone's, and for the largest customers dedicated capacity. The same thinking applies to internal callers: a batch job and an interactive API should not share a pool.

Operating bulkheads

Expose per-pool utilisation and rejection counts; a pool at its limit is an early warning for its dependency. Alert on rejections for critical compartments. Review pool sizes when traffic patterns change. Resist the temptation to raise a saturated pool's limit during an incident without understanding why it saturated; that usually just moves the failure to the next shared resource.