Requests, Responses and Latency
- Pradeep P
- 3 days ago
- 4 min read
Layer 1 · Post 2 of 15
← Previous: What Actually Makes a System "Distributed"? → Next: Horizontal vs Vertical Scaling
Layer 1 — The building blocks · Post 2 of 88
Every user action is a request that waits for a response. Latency is that wait, and it compounds as a request hops through more services.
What you'll learn
What a request/response actually is on the wire
Why latency adds up across hops, and why "average" hides the pain
How to talk about p50, p99, and timeouts without sounding vague
The idea in one minute
A request is a message that says "please do this." A response is the message that says "here is the result" (or "I failed").
Latency is the time between sending the request and getting the response.
Users feel latency as lag. Systems feel it as occupied resources: a thread, a connection, and a slot in a queue are all held until the response comes back. Slow is not only annoying. Slow is expensive.
Why it matters
System design interviews often start with "how many requests per second?" That number is useless without latency.
If each request holds a server for 10 ms, one machine might handle thousands per second. If each request waits 2 seconds on a downstream database, the same machine can handle far fewer — its workers are busy waiting.
Latency also compounds. A page that looks like one click is often many requests:
Browser → API → Auth → User service → Database
\→ Recommendations → Cache
The user waits for the slowest path that the page needs. Five "fast" services in a row can still feel slow.
How it works
A typical HTTP request looks like this:
The client opens (or reuses) a connection.
It sends headers and a body: method, path, data.
The server does work: CPU, disk, other network calls.
The server sends a status and a body back.
The client reads it and continues.
Latency is the sum of several pieces:
Piece: Network; What it is: Packets traveling, plus TLS and handshakes; Typical scale: 1 ms in a datacenter, 50–200+ ms across continents
Piece: Queueing; What it is: Waiting for a free worker or connection; Typical scale: Near zero until you are busy, then it spikes
Piece: Compute; What it is: CPU actually doing work; Typical scale: Microseconds to milliseconds, unless you wrote a hot loop
Piece: Downstream; What it is: Waiting on other services; Typical scale: Often the largest piece
The important mental model: most of a request's life is waiting, not computing.
Averages lie
If 99 requests take 20 ms and 1 request takes 5 seconds, the average is still ~70 ms. That looks fine on a dashboard. The one user in a hundred had a terrible day.
Engineers talk in percentiles:
p50 (median): half of requests were faster than this.
p99: 99% were faster; 1% were slower. This is where timeouts, retries, and angry tweets live.
Design for the tail, not the average.
A simple example
You open a profile page. The API does three things in sequence:
Fetch the user (20 ms)
Fetch their posts (80 ms)
Fetch follower count (30 ms)
Total: 130 ms, plus network to the browser.
If you run the three fetches in parallel, the page waits for the slowest one: 80 ms. Same work, less waiting. That is why fan-out and parallelism show up in every "design X" discussion.
Now add a cache. If the posts are already in Redis (2 ms) most of the time, p50 looks great. The first request after a cache miss still pays 80 ms. p99 depends on how often you miss.
Common mistakes
Optimizing CPU when you are waiting on I/O. Profiling the app server will show it "idle" while the database is the real bottleneck.
Ignoring the client. A 50 ms API in Virginia feels like 200 ms in another country. CDNs and edge caches exist for this (later posts).
Setting one global timeout of 30 seconds. Users have already left. Timeouts should be tight at each hop so a dead dependency fails fast.
Measuring only inside the server. The user felt DNS, TLS, and the last mile. Your /health latency is not their latency.
How this shows up in real systems
APIs publish SLOs like "p99 under 200 ms." That is a latency budget. Every new hop has to fit inside it.
Mobile apps batch requests or use a backend-for-frontend so a weak network does not pay for 20 round trips.
Streaming (video, LLM tokens) changes the game: first-byte latency matters more than "wait for the entire response."
When someone asks "is this design fast enough?", translate that to: what is the request path, what does each hop cost, and what does p99 look like when something is slow?
Recap
Almost every user action is request in, response out.
Latency is the wait. It adds across hops and hides in the tail (p99).
Fast systems reduce hops, parallelize independent work, and stop waiting with timeouts.
Next we look at the first lever people reach for when latency and load get worse: making the machine bigger, or adding more machines.
Layer 1 · Post 2 of 15
← Previous: What Actually Makes a System "Distributed"? → Next: Horizontal vs Vertical Scaling
Comments