Retries

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 3 — Reliability

Layer 3 · Post 2 of 14

← Previous: Timeouts → Next: Exponential Backoff

Layer 3 — Reliability · Post 27 of 88

Retries recover from transient failures. Done badly, they turn a small outage into a stampede.

What you'll learn

Which failures are safe to retry, and which you must not
Why retries without a budget and a cap cause retry storms
How retries assume idempotency — and why "just retry" is an incomplete design

The idea in one minute

A retry is a second (or third) attempt after a failure you believe is temporary: a timeout, a connection reset, a 503, a packet loss.

Call payments --> timeout | +--> wait a bit --> call again --> 200 OK

You already paid for timeouts. Retries are how you turn "that one blip" into a success the user never sees.

They are also how a 5-second payments blip becomes 3× the traffic just as payments is trying to stand up. Every client retries at once. That is a retry storm.

Why it matters

Networks and multi-tenant neighbors fail in short bursts. Refusing to retry means users eat every blip. Retrying blindly means you amplify load on the sick service — the opposite of Layer 3's job, which is containing failure.

Webhooks (Post 25) retry for days. SQS retries until the message is old. Your HTTP client retries in milliseconds. Same idea, different blast radius.

Interview: say what you retry, how many times, from where (client vs worker), and whether the operation is idempotent. "We retry" without those is a red flag.

How it works

Retry only what is transient.

Safe-ish: connect failures, resets, timeouts if you never saw a response, 429 (with Retry-After), 503, 502 from a proxy that never reached the app.

Do not retry: 400, 401, 403, 404, 409 (usually), most 422s. The second call will fail the same way. Do not retry a 200 with a business error in the body.

Ambiguous: timeout after you sent a POST. The server might have charged the card. You cannot blindly retry unless the POST is idempotent (next post after backoff).

Limits you actually set:

Max attempts (often 2–3 on the request path, more in a queue worker).
A total deadline inherited from the timeout budget. Three 200 ms tries need ~600 ms plus gaps — that must still fit the gateway.
Retry only at one layer when you can. If the SDK, the mesh, and your code all retry, 3×3×3 = 27 calls.

Where to retry:

Synchronous request path: few, fast, only for idempotent or clearly unsent calls.
Queue / webhook worker: many, with backoff, then a dead-letter queue. The user is not holding a connection.

You will add exponential backoff next so the retries are not a synchronized hammer.

A simple example

The API fetches a user's cart from Redis. The call times out (200 ms). Retrying once is reasonable: Redis blips, the data is a read, doing it twice is safe.

The same API charges a card via Stripe. The HTTP client times out. You do not fire the same charge again from the request thread unless you sent an idempotency key. You persist "charge pending" and let a worker finish it, or you retry with the same key. Otherwise the user sees one spinner and two charges.

Common mistakes

Retrying every HTTP error. 400s and 404s are not transient.

Retries on the hot path with no cap. A timeout cascade: A retries B, B retries C, traffic explodes.

Retrying after a timeout on a write. You do not know if it committed. That is an idempotency problem, not a "try again" problem.

Client, sidecar, and library all retrying. Count the multiplier.

No dead-letter. Infinite retries on a poison message (bad JSON) burn CPU forever.

How this shows up in real systems

AWS SDKs, Google Cloud clients, Stripe: retries with backoff built in; you still configure max attempts.
SQS / RabbitMQ / Kafka consumers: at-least-once delivery is retries. Consumers must be idempotent.
Kubernetes: CrashLoopBackOff is retries with backoff for processes.
Envoy / Istio retry policies: per-route, with retry-on conditions (connect-failure, 5xx, reset) so you do not retry 404s.

Recap

Retry transient failures; do not retry logic errors.
Cap attempts and amplification — storms make outages worse.
Writes need idempotency before retries are safe. Backoff decides when the next try happens.