Timeouts

Pradeep P

Aug 31

3 min read

Series: Modern System Design · Layer 3 — Reliability

Layer 3 · Post 1 of 14

← Previous: Webhooks → Next: Retries

Layer 3 — Reliability · Post 26 of 88

A timeout is how you stop waiting. Without one, a slow dependency can stall your whole system.

What you'll learn

Why "wait forever" is a production bug, not a default
How to size a timeout from the caller's budget, not a round number
The difference between connect, request, and idle timeouts — and why they must nest

The idea in one minute

A timeout is a deadline. You call another service, a database, or a disk. If the answer is not back by time T, you stop waiting and treat it as failure.

You --request--> Payments v abort, free the worker, maybe retry later

Without a timeout, a hung TCP connection holds a thread, a connection slot, and a user. One slow dependency becomes your outage.

Layer 2's webhooks already assumed this: if their URL never returns, you cannot wait all day. Layer 3 starts here because every later tool (retries, breakers, failover) needs a clock.

Why it matters

Latency from Post 2 is not just "slow." Slow occupies capacity. If each request holds a worker for 30 seconds instead of 50 ms, throughput collapses by orders of magnitude. You run out of threads, file descriptors, and Postgres connections while CPU looks idle.

Interviews: if you draw Service A calling Service B and never mention a timeout, the design hangs on B's worst day. A 200 ms cutoff on a 250 ms p99 also looks like a flaky network. Size them on purpose.

How it works

Pick a budget from the outside in.

The user (or the API gateway) has a budget — say 2 seconds for checkout.
Your service must finish inside that, including its own work.
Each downstream call gets a slice smaller than what remains — a bit above that call's p99, with room for one retry only if you planned one.
Nested timeouts must get tighter as you go in. If the gateway waits 2 s, your service cannot wait 5 s on the database. The outer caller already hung up.

Set all of: connect (black-holed hosts fail in tens of ms), request/deadline (the whole RPC — gRPC deadlines live here), idle/read (bytes stop mid-response), and server-side (statement_timeout in Postgres so a runaway query cannot outlive the client).

A timeout is a local decision. You do not know if the other side finished. That is why the next posts exist: retries and idempotency.

A simple example

Checkout calls payments. Payments p50 is 40 ms, p99 is 180 ms, p99.9 is 900 ms.

You set the client timeout to 250 ms. Most calls succeed. The ugly tail fails fast. The user sees "try again" instead of a spinning button. Your thread pool stays available for the next checkout.

If you set 30 seconds "to be safe," 50 hung payment connections fill the pool. Healthy users cannot pay either. The timeout was supposed to protect you, not to be polite to a dead peer.

Common mistakes

No timeout, or the language default (often none, or minutes). HTTP libraries and ORMs are optimistic. You must set this.

One 30-second timeout everywhere. Users left. Gateways already cut you off. Inner calls should be shorter.

Timeout longer than the caller's timeout. You keep working after the client is gone, then maybe retry, then maybe double-charge. Match the chain.

Only timing out the client. The database query continues. Set server-side limits too. Timeouts belong near p99, not p50, or you retry a healthy-but-slow service into the ground.

How this shows up in real systems

Envoy / nginx / AWS ALB: timeout, idle timeout, and per-try timeout are first-class. Envoy's per-try timeout is how you retry inside a larger budget.
gRPC deadlines: a timestamp that propagates. Downstream sees how much time is left.
Postgres statement_timeout, Redis socket timeouts, Stripe / GitHub API clients: every mature SDK asks you to configure this.
Kubernetes probes: a probe with no timeout is how a dead pod stays "ready."

Recap

A timeout is a deadline so a slow or dead dependency cannot hold your resources.
Size from the user's budget inward; inner timeouts must be tighter.
Timeouts make failures visible. Next you decide whether to retry them.