Little's Law

Pradeep P
2 days ago
4 min read

Series: Modern System Design · Layer 1 — The building blocks

Layer 1 · Post 17 of 21

← Previous: Fallacies of Distributed Computing → Next: Amdahl's Law

Layer 1 — The building blocks · Post 90 of 119

In a stable system, average items in the system equals arrival rate times average time in the system.

What you'll learn

The formula L = λW, in words you can use at a whiteboard
Why adding servers does not help if each request still waits in a long queue
How this ties to latency, thread pools, and "the site is up but everything is slow"

The idea in one minute

Little's Law (John Little, 1961): in a system that is stable (not growing forever),

L = λ × W

L — average number of items in the system (in flight: queued + being served)
λ (lambda) — average arrival rate (requests per second)
W — average time in the system (latency, including queue time)

Arrivals λ  -->  [ queue + workers ]  -->  done
                  L items live here
                  each spends W seconds

If 100 requests/s arrive and each stays 0.2 s, you have about 20 requests inside the system at any moment. Those 20 occupy threads, connections, memory, and locks.

Why it matters

Post 2 talked about latency. Little's Law tells you why latency and concurrency are the same problem.

You cannot have high arrival rate, high latency, and a small server. The product L has to live somewhere: goroutines, Tomcat threads, DB connections, Kafka consumer lag. When L exceeds what you provisioned, new arrivals queue. Queueing makes W worse. Worse W makes L bigger. That is the melt-down loop people call "the site is up, but nothing loads."

Interviews: they want you to size concurrency from RPS × latency, not from a round number you like.

How it works

Stability first. If λ is higher than the service can finish, L grows without bound. Little's Law describes the steady state, not a death spiral. Capacity must be ≥ arrival rate over the window you care about.

W includes waiting. People quote "p50 handler time is 10 ms" and provision for that. Users feel queue + handler. If the queue is 200 ms, W is 210 ms, and L is 21× larger than the handler-only estimate.

You can pick two. Want lower L (fewer connections)? Drop λ (shed load) or drop W (faster work, more parallelism on the slow part). Want higher λ at the same W? You must allow higher L — more workers, more connections — or W will rise.

Utilization. As workers approach 100% busy, queues explode. Little's Law does not need a specific queueing model (M/M/1 etc.) to be true, but those models explain why W jumps near saturation. Run with headroom.

Amdahl (next post) limits how much you can shrink W by adding parallelism. Little's Law tells you what happens to L if you fail.

A simple example

Checkout API: 500 RPS, average 200 ms end-to-end.

L = 500 × 0.2 = 100 in-flight requests.

Each request holds a Tomcat thread and a DB connection for most of that 200 ms. You need on the order of 100 threads and 100 connections just for the average, plus burst. A pool of 20 connections means the extra 80 wait. Waiting raises W. L grows. Timeouts start (Post 26).

Fix options, all Little's Law:

Make W smaller (index, cache, stop N+1).
Lower λ on this path (rate limit, async the slow part onto a queue).
Raise the pool so L can actually fit — knowing that just raises occupancy, not magic.

If you "scale out" to 5 app servers but the database still has 20 connections total, L still cannot fit on the bottleneck. Horizontal scaling of the wrong tier does not change W at the lock.

Common mistakes

Using p50 as W. Tail latency (p99) is what fills the pools. Size from a high percentile, then add margin.

Counting only "running" work. Items in the queue are in the system. Kafka consumer lag is L for that pipeline.

Ignoring Little's Law when load testing. 10k RPS at 2 s latency means 20,000 in flight. Your client and server will fall over from concurrency, not from CPU graphs looking "fine" at the start.

Thinking more RAM fixes W. Memory holds L. It does not shrink time in the system unless the slowness was paging.

How this shows up in real systems

Thread / goroutine / lambda concurrency limits: a cap on L. Hit it and you queue or reject.
DB max_connections: a cap on L at the datastore.
Kafka consumer lag: L in the log. λ is produce rate; W is time until consume.
Autoscaling on CPU: often too late. Scaling on in-flight or queue depth is scaling on L.

Recap

L = λW: in-flight work is rate times latency.
Slow plus busy means huge L, which means pool exhaustion and then worse latency.
Size concurrency from this product; do not guess a thread count.
Next: why extra machines still hit a wall — Amdahl's Law.