Bulkheads

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 3 — Reliability

Layer 3 · Post 6 of 14

← Previous: Circuit Breakers → Next: Rate Limiting

Layer 3 — Reliability · Post 31 of 88

Bulkheads isolate failure. One overloaded pool, tenant, or dependency should not be allowed to sink the rest of the ship.

What you'll learn

Isolation as a resource problem: threads, connections, and queues
How to partition pools per dependency or tenant without over-fragmenting
How bulkheads differ from circuit breakers (limit vs stop-calling)

The idea in one minute

Ships have bulkheads: watertight walls. A hole in one compartment does not flood the hull.

In software, the "water" is shared capacity — a thread pool, a DB connection pool, a queue, a Kubernetes CPU limit. If recommendations saturates the only pool, checkout waits for a free worker and dies too.

[ API process ] pool A (max 20) --> Payments (slow is contained) pool B (max 20) --> Recs (on fire, only pool B fills) pool C (max 50) --> Postgres (checkout still has C)

A circuit breaker stops calling. A bulkhead caps how much of you a dependency can consume even while you still call it.

Why it matters

Timeouts bound duration. Breakers bound whether you call. Bulkheads bound concurrency. You need all three. A dependency that is slow-but-not-failing (p99 2 s, still 200 OK) may never trip a breaker if your error threshold is "5xx." It will still pin every thread. That is the classic "death by slow downstream."

Interviews: "noisy neighbor" and "blast radius" are bulkhead words. Multi-tenant SaaS that shares one DB pool per pod is a bulkhead design, whether you used the name or not.

How it works

Pick the scarce resource and split it.

Per dependency: separate HTTP client pools, gRPC executors, or Hystrix/Resilience4j bulkhead sizes. Recs can use 10 concurrent calls; payments 30; never 40+40 from a pool of 40.

Per tenant / customer: separate queues, rate limits (next post), or even databases for the huge customer. One customer's import job cannot occupy all workers.

Per workload: interactive vs batch. Batch analytics gets its own replica and its own thread pool so a report cannot eat checkout connections.

When the bulkhead is full, fail that compartment (reject, 503, skip the optional call). Do not borrow from the next pool.

Size from concurrency ≈ arrival_rate × latency. Recs at 100 rps × 200 ms needs ~20 slots. Too small you reject healthy traffic; too large the wall is fake. Kubernetes limits and separate Deployments are the same pattern at the platform layer.

A simple example

One Node/Java service talks to Redis (cache), Postgres (orders), and a Python ML recs service.

Shared pool of 50 threads. Recs p99 jumps to 3 s. All 50 threads sit in recs HTTP calls. Postgres is fine. Checkout cannot get a thread. Site-wide outage.

Split: 15 threads max to recs, 20 to Postgres, 10 to Redis, 5 for everything else. Recs widgets empty or error. Checkout still commits orders. You have a recs incident, not a company incident.

Common mistakes

One giant executor for all outbound I/O. Looks simple in the framework default.

Tiny pools that you then enlarge in prod until they are one pool again. Review the numbers when latency changes.

Isolating CPU but sharing the database. The bulkhead moved; Postgres max_connections is now the hull.

Fairness without isolation. A global FIFO queue still lets one tenant submit 100k jobs first.

Calling this a circuit breaker. Breaker: stop sending. Bulkhead: only N in flight. Use both.

How this shows up in real systems

Resilience4j Bulkhead / Hystrix thread pools: the library form.
NGINX / Envoy connection limits per upstream cluster.
Postgres roles, PgBouncer pools, RDS Proxy: connection bulkheads.
Kubernetes namespaces, ResourceQuotas, separate Deployments; AWS accounts and SQS queues per tenant.
Kafka consumer groups isolated per workload so a lagging analytics consumer does not block the serving path.

Recap

Bulkheads cap shared resources so one failure cannot take all capacity.
Split pools by dependency, tenant, or workload; reject when full.
Complements timeouts and breakers. Next: rate limiting — a bulkhead at the front door.