LLM Request Queues
Series: Modern System Design · Layer 6 — Modern systems
Layer 6 · Post 14 of 26
← Previous: Token Streaming → Next: GPU Scheduling
Layer 6 — Modern systems · Post 76 of 88
LLM request queues absorb bursts, enforce fairness, and keep GPUs busy without overloading them into timeouts.
What you'll learn
Why you put a queue in front of GPUs instead of letting every HTTP request land on a model
Admission control, fairness, and what "queue wait" vs "generation time" means to the user
How to shed load with 429s before you melt KV cache and miss SLOs
The idea in one minute
An LLM inference server has a hard capacity: GPU memory, batch slots, tokens per second. HTTP does not. Users arrive in bursts. If you forward every request straight to the GPU, you get timeouts, OOM, and a thundering herd of retries.
A request queue sits in front of the workers:
Clients | v [ API / gateway ] | v [ Queue: wait, priority, admit or reject ] | v [ Inference workers / GPUs ]
The queue's job is not "store messages forever." It is backpressure with a policy: who waits, who jumps the line, and who gets a fast no.
Why it matters
GPUs are expensive and sticky. A request that is "in flight" holds KV cache until it finishes or is cancelled. You cannot scale them the way you scale stateless API pods.
Without a queue you will:
Over-admit. Ten extra concurrent generations can blow memory and kill healthy work.
Starve interactive users. A batch job of 200 summaries sits on the same GPUs as "type a sentence."
Lie about latency. The user waited 40 seconds in a hidden Nginx backlog, then 2 seconds generating. Your p99 looks like the model is slow. It is not.
Interviewers will ask how you handle a traffic spike on a chatbot. "Add more GPUs" is the expensive half. The other half is not accepting work you cannot finish in the SLO.
How it works
Two clocks
Measure both, or you will debug the wrong layer:
Queue wait: time from accept until a worker starts prefill.
Generation time: prefill + decode (and tool rounds, if this is an agent).
Users feel the sum. Your autoscaler should watch wait, not only GPU utilization.
Admission control
Before enqueue:
Estimate cost (prompt tokens, max output, maybe "this tenant is on the huge model").
Check in-flight work against a concurrency cap (often tighter than "RPS").
Admit, queue, or reject (HTTP 429 / 503 with Retry-After).
A bounded queue (depth + max wait) is a feature. An unbounded queue is a timeout factory.
Fairness and priority
A single FIFO is simple and unfair. Typical split:
Interactive (chat, low max tokens) vs batch (overnight jobs).
Per-tenant weights so one customer cannot own the fleet.
Deadlines: drop or deprioritize if wait already exceeds the SLO. Do not start a 30-second generation that will be cancelled.
Priority queues still need starvation protection, or batch never runs.
Cancellation
Users hit stop. The client disconnects. If you do not cancel the in-flight generation, you pay GPU time for tokens nobody will read, and you hold a batch slot. Wire abort from the HTTP layer into the worker.
A simple example
Your product has 4 GPUs. Each can hold about 16 concurrent chats given your context sizes. At 10:00 a marketing email lands and 400 people open the assistant.
Without a queue: all 400 hit the workers. KV cache explodes. Existing chats stall. Half the new users retry. You now have 800 requests.
With a queue: you cap in-flight at 64. The next 80 wait with a visible "busy" state and a position or ETA. The rest get 429 and a retry-after of a few seconds. Interactive stays on a high-priority lane. A nightly embedding job is paused. GPUs stay full, not panicked.
Common mistakes
Treating the LLM like a normal REST service and sizing on RPS. One request can last 60 seconds and occupy a GB of GPU memory. Concurrency is the unit.
Unbounded Redis / SQS "so we never drop." You drop in the user's browser instead, after they waited a minute. Bound the queue. Fail fast.
One FIFO for chat and batch. Batch wins on volume. Chat dies. Separate lanes.
Ignoring disconnects. Cancel work. Count cancelled tokens; they still cost you on self-hosted GPUs.
Autoscaling only on GPU util. Util can be 90% while wait is 20 seconds. Scale (or shed) on queue wait.
How this shows up in real systems
vLLM / TensorRT-LLM / TGI: waiting queues plus continuous batching; you still want a product-level queue in front for multi-tenant fairness.
OpenAI, Anthropic, Bedrock: 429s and token-per-minute limits are their queue, pushed to you. You still need your own for fan-out and fairness.
Celery / SQS for batch: fine for offline jobs; do not put interactive chat only on a minute-scale consumer.
API gateways: concurrency limits and retry budgets belong here, not only in the Python process.
Layer 1's load balancer spreads connections. This queue spreads scarce GPU time.
Recap
A queue in front of inference is admission control, not a mailbox.
Track wait and generation separately; cap concurrency, not just RPS.
Fairness, priorities, bounded depth, and cancel-on-disconnect keep GPUs busy without lying to users.
Next is how those GPUs actually pick which request, batch, and model to run.
Series: Modern System Design · Layer 6 — Modern systems
Layer 6 · Post 14 of 26
← Previous: Token Streaming → Next: GPU Scheduling
Comments