GPU Scheduling

Pradeep P
3 days ago
4 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 15 of 26

← Previous: LLM Request Queues → Next: Model Routing

Layer 6 — Modern systems · Post 77 of 88

GPU scheduling decides which model, which batch, and which request gets the scarce accelerators next. Waste here is measured in dollars per minute.

What you'll learn

Why GPU time is scheduled in batches, not "one request per core"
Prefill vs decode, KV cache, and why memory — not FLOPs — often gates you
How continuous batching and multi-model packing trade latency for dollars

The idea in one minute

A GPU wants wide, regular work. Autoregressive decoding is the opposite: each new token needs the previous one, and every live request holds a growing KV cache.

GPU scheduling is the policy that answers: whose tokens run in this kernel, which sequences join the batch, and whether you swap models.

Queued requests | v [ Scheduler ] | | \ pack into a batch v v [ Prefill ] [ Decode step for N live sequences ] | v GPU kernels + KV cache in HBM

Idle SMs while you wait for one long prompt is how you light money on fire.

Why it matters

Cloud GPUs are billed by the minute whether you emit tokens or sit at 12% utilization. Memory is worse: one 70B model plus long contexts can fill an H100. You do not "spin up another thread."

If you skip this in an interview, you sound like you designed a CPU web farm. The follow-up is always: how do you keep the GPU full without blowing p95 latency?

Scheduling is also how you share one box across models (chat, embed, rerank) without thrashing weights in and out of HBM.

How it works

Prefill vs decode

Prefill reads the prompt and builds KV cache. It is compute-heavy and can be batched across sequences with similar lengths.

Decode emits one token at a time. It is often memory-bandwidth bound: you stream weights and cache more than you do huge matmuls. Many live sequences in one step (continuous batching) amortize the weight read.

A naive "finish request A completely, then start B" leaves the GPU half empty during decode. Modern servers add and remove sequences every step.

The real constraint is memory

Each live request occupies:

Model weights (shared)
KV cache (per request, grows with context + generated tokens)
Activations for the current batch

The scheduler's first question is not "is the GPU busy?" It is does this request fit? If you admit a 128k-context job, you may have to preempt or refuse smaller chats. Paged KV (vLLM-style) makes fragmentation less stupid; it does not create free HBM.

Policies you actually pick

Throughput: large batches, higher token/s, worse time-to-first-token.
Latency SLO: smaller batches, maybe reserved slots for interactive.
Preemption: pause a long decode to run a short one; costs a KV shuffle.
Model affinity: keep one model hot. Swapping a 70B checkpoint is a deploy, not a context switch. Prefer MIG, multi-LoRA, or one model per replica.

A simple example

You serve an 8B chat model on one A100. Ten users send 20-token questions. Prefill is quick. Decode runs as a batch of 10: each step produces 10 tokens in roughly one weight-stream. Token/s looks great.

Then one user pastes a 30,000-token PDF. Prefill is huge. KV cache eats room for six of the chats. If you naively start it, the other nine stall or OOM.

A sane scheduler: estimate cache, limit max concurrent tokens, maybe run the PDF on a "long context" pool, and keep a few decode slots for short chats. Utilization stays high; the product still feels interactive.

Common mistakes

One-request-at-a-time because that is how you wrote the demo. You pay 10× on decode.

Batching only at the HTTP layer ("wait 50 ms then send 8 prompts"). Continuous batching at the iteration level is the real win.

Ignoring KV size in admission. You schedule until CUDA OOM, then everything dies.

Swapping full models every request to be "multi-model." Load the hot model and route the rest (next post) or use LoRA / separate replicas.

Optimizing FLOPs while HBM is full. Profile memory and cache hit of the scheduler, not only TFLOPS.

How this shows up in real systems

vLLM: paged attention, continuous batching, a waiting vs running queue.
TensorRT-LLM, TGI, Triton: same packing problem, different kernels and batching knobs.
Kubernetes device plugins: they schedule pods onto GPUs. They do not pack sequences. You still need an inference scheduler inside the process.

The request queue (previous post) decides who is allowed to wait. This layer decides how their tokens share the silicon.

Recap

GPUs want batched, regular work; decode wants many live sequences per step.
KV cache is the scarce resource; admit and preempt with memory in mind.
Continuous batching, isolation (MIG / replicas), and not swapping 70B weights on every call are how you spend less per token.

Once you can run models efficiently, you still have to pick which model a request deserves.

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 15 of 26

← Previous: LLM Request Queues → Next: Model Routing