Head-of-Line Blocking

Pradeep P
2 days ago
3 min read

Series: Modern System Design · Layer 2 — Communication

Layer 2 · Post 13 of 14

← Previous: Poison Messages → Next: Backends for Frontends

Layer 2 — Communication · Post 97 of 119

When the first item in a queue or connection is slow, every later item waits even if they are ready.

What you'll learn

HOL blocking on a TCP connection, an HTTP/1.1 pipeline, and a FIFO queue
Why HTTP/2 and HTTP/3 were invented, and what they still do not fix
How to keep one slow customer from stalling everyone else

The idea in one minute

Head-of-line (HOL) blocking: a shared line, one slow item at the front, everyone behind waits — even if their work is small and ready.

[ BIG SLOW JOB ][ tiny ][ tiny ][ tiny ]
        ^
        only this is running
        tinies are blocked by position, not by cost

Classic places:

HTTP/1.1: one outstanding request per connection (or pipelining, which still stalls on the first slow response).
TCP: a lost packet holds later packets in that connection until it is retransmitted — even if those later packets already arrived.
Queues / Kafka partitions / mutexes: one poison or huge message (previous post) blocks the partition.

Why it matters

Latency SLOs die from HOL more often than from average work being hard. Little's Law: the tinies sit in the system (L up, W up) because they are queued behind a stranger.

Interviews: "why HTTP/2?" — multiplexed streams so one slow image does not block CSS. "Why not one Kafka partition?" — throughput and HOL. Bulkheads (Post 31) are HOL prevention for thread pools.

How it works

More independent lines. HTTP/1.1 browsers opened ~6 connections per host to dodge HOL. HTTP/2 multiplexes streams on one TCP connection — application HOL mostly gone; TCP HOL remains (a lost packet still stalls all streams). HTTP/3 / QUIC: streams on UDP so a lost packet stalls one stream, not all.

Don't share a FIFO for unlike work. Interactive API calls and a 2 GB export should not share one worker thread or one SQS queue without priority. Use separate queues, priority queues, or a lane for "small / deadline."

Partition isolation. Kafka: one slow consumer group member holds one partition. Other partitions proceed. That is better than one global queue — unless all hot keys hash to that partition (Post 94).

Timeouts and cancellation. A stuck head that never finishes is HOL forever. Bound it (Post 26). Preempt or DLQ.

Fairness. Token buckets per tenant so one customer cannot occupy the only worker. That is HOL as a noisy neighbor problem.

A simple example

A mobile app uses HTTP/1.1 to one API host. The first request is "generate PDF" (8 seconds). Requests 2–4 are "fetch unread count" (20 ms each). They wait 8 seconds because they are on the same connection pipeline, or because a single-threaded proxy handles one request at a time.

HTTP/2: unread count streams complete while the PDF stream still runs (unless a TCP loss stalls the whole connection). Better: don't put PDF generation on the user-facing connection at all — enqueue it, return a job id (async, Post 18).

At the queue: if PDF jobs and unread-count jobs share one FIFO worker pool, the same HOL happens without HTTP in the story.

Common mistakes

One "processing" queue for all event types. A 10-minute video transcode sitting in front of password-reset emails.

Assuming HTTP/2 ended HOL. TCP still couples streams. Packet loss on mobile still hurts; that is why QUIC exists.

Infinite concurrency to "avoid HOL." You just move HOL to the database (lock convoy) or explode L (Little's Law). You want lanes, not infinite workers.

FIFO "for fairness" without size limits. Fair among messages, unfair among users when one user sent a monster.

How this shows up in real systems

HTTP/1.1 vs HTTP/2 vs HTTP/3: the textbook HOL story.
gRPC over HTTP/2: many RPCs, one connection; still TCP HOL; also a slow handler can fill a stream window.
SQS FIFO / Kafka partition / RabbitMQ single active consumer.
Head-of-line in switches: classic networking; same shape as your thread pool.

Recap

HOL: position in a shared line beats actual cost.
Split lanes (protocol streams, queues, partitions, bulkheads); bound the head.
Next: a named API pattern so clients are not stuck in one slow, one-size-fits-all backend — BFF.