Circuit Breakers

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 3 — Reliability

Layer 3 · Post 5 of 14

← Previous: Idempotency → Next: Bulkheads

Layer 3 — Reliability · Post 30 of 88

A circuit breaker stops calling a failing dependency for a while so you fail fast locally instead of piling on a service that is already down.

What you'll learn

Closed, open, and half-open — what each state does to traffic
Why breakers exist when you already have timeouts and retries
What to do instead of calling (fail, cache, fallback) — and when fallbacks lie

The idea in one minute

A circuit breaker watches calls to one dependency. If too many fail (timeouts, 5xx), it opens: further calls fail immediately without hitting the network.

Closed --too many failures--> Open --cooldown--> Half-open --probe ok--> Closed --probe fails--> Open again

Named after electrical breakers: stop sending current through a short. Timeouts still wait. Retries still hit the victim. A breaker sheds load so payments can recover while you return a fast error (or a degraded path).

Why it matters

Timeouts + retries without a breaker: every user request still occupies a worker for timeout × attempts against a dead host. You exhaust your thread pool (bulkheads next) and you keep the dependency from coming back — every recovered instance is slammed by the waiting crowd.

Interviews: after you mention retries, they will ask "what if it stays down?" Breaker is the answer, plus a product decision: error, stale cache, or skip that widget.

How it works

Closed (normal): calls go through. You count successes and failures in a window (last N calls, or last 10 seconds).

Open: error rate or consecutive failures crossed a threshold (e.g. 50% of 20 calls, or 5 in a row). Calls fail fast — CircuitOpen — for a cooldown (say 5–30 s).

Half-open: allow a few probes (1–few requests). If they succeed, close. If they fail, open again. This is how you detect recovery without a stampede.

Tune:

What counts as failure (timeouts yes; 404 no — that is not "payments is down").
Window size: too small and one blip opens you; too large and you are slow to protect.
Cooldown: too short and you flap; too long and you stay degraded after recovery.

Scope the breaker per dependency, often per host. One bad replica should not black-hole the whole payments fleet (Envoy outlier detection is this idea). Combine with retries: retry other hosts, not the same open circuit.

When open, pick a fallback deliberately: error the user, serve last-known from Redis, skip recommendations. A fallback that returns "stock: 0" or "price: 0" is a new incident.

A simple example

The homepage calls recommendations. Recs is on fire (p99 5 s, 80% 500s).

Without a breaker: every page load waits on recs, homepage p99 dies, checkout shares the same worker pool and dies too.

With a breaker: after the threshold, homepage skips recs, shows the rest of the page in 50 ms. Recs gets a cooldown. After half-open probes succeed, widgets come back.

Checkout must not fallback-fake a payment success. Fail fast and tell the user.

Common mistakes

One breaker for the whole process. A sick recs service trips a global switch and you stop calling auth.

Counting 4xx as failures. You open the circuit because of bad client input.

No half-open. You either hammer forever or stay open until a restart.

Fallback that looks like success. Cached "authorized" or empty cart as if it were live data.

Breaker instead of a timeout. If you never time out, the breaker never sees failures — it sees stuck calls. Timeouts feed the breaker.

How this shows up in real systems

Netflix Hystrix (historical) and Resilience4j, Polly (.NET): library breakers around HTTP/gRPC clients.
Envoy outlier detection and retry budgets: mesh-level "stop calling that host."
AWS ALB / API Gateway plus alarms: cruder, but same idea — stop sending when targets are unhealthy.
Stripe / GitHub client guidance: fail fast on outages; do not retry storms (Posts 27–28).

Recap

A breaker stops calling a failing dependency so you fail fast and it can recover.
States: closed → open → half-open.
Pair with timeouts; isolate per dependency. Next: bulkheads so even waiting calls cannot sink the ship.