What Is an LLM Inference Server?

Pradeep P
3 days ago
4 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 11 of 26

← Previous: Designing a File Storage System → Next: Streaming LLM Responses

Layer 6 — Modern systems · Post 73 of 88

An LLM inference server loads a model and runs generation requests. It is the new 'app server', except the unit of work is tokens on a GPU.

What you'll learn

What an inference server actually holds in memory (weights, KV cache, running batches)
How prefill vs decode, and continuous batching, shape latency and throughput
Where vLLM, TGI, TensorRT-LLM, and "just call OpenAI" sit in a design interview

The idea in one minute

An LLM inference server is a process (or fleet) that has loaded model weights onto a GPU (or several) and exposes an HTTP/gRPC API: prompt in, tokens out. Your product API does not import transformers in the request path on a CPU box and hope. It calls this server the way it would call a payment PSP — except the scarce resource is GPU memory and time, not a bank.

Product API / gateway | v [ Auth, quota, prompt assemble ] | v [ Inference server: weights on GPU ] | prefill prompt --> KV cache | decode token-by-token (batched with other reqs) v [ Stream or JSON back to client ] | OOM / queue full? 429 or shed; never silently drop on the GPU thread

The unit of work is a token. A request can last seconds. That is unlike a 10 ms user-service call.

Why it matters

If you treat GPUs like stateless app replicas, you will OOM, underutilize the chip, or timeout the user. Interviewers for "AI features" now expect you to name inference as a service, batching, and KV cache — not only "we'll use GPT-4."

Cost and latency live here. The next posts (streaming, queues, GPU scheduling) all hang off this box.

How it works

Load. At boot, the server reads shards of weights into GPU RAM (and CPU/NVMe if offloading). A 70B model in 16-bit needs tens of gigabytes — often tensor parallel across 2–8 GPUs. You do not reload weights per request.
API. OpenAI-compatible POST /v1/chat/completions is the common contract: messages, max_tokens, stream. The server tokenizes the prompt (CPU), then runs the model.
Prefill vs decode. Prefill: process the whole prompt, fill the KV cache (attention keys/values per layer). Compute-heavy, parallel over sequence. Decode: emit one token, append to cache, repeat. Memory-bandwidth heavy. Time-to-first-token is dominated by prefill; the rest of the wait is decode.
Batching. GPUs like large matrix multiplies. Continuous batching (vLLM-style) lets new requests join a running batch at decode steps so you do not wait to form a static batch of 32. Throughput goes up; per-user latency can still be OK if you cap batch size.
Memory. KV cache grows with batch × layers × seq_len. Long context + many concurrent users is the OOM. Paged KV (vLLM) treats cache like virtual memory so fragmentation does not waste the GPU.
Failure. GPU crash: the process dies; the load balancer marks the replica unhealthy. In-flight generations abort — clients retry or resume (hard). Overload: queue (post 76) or 429. Do not accept unbounded concurrency "because HTTP is cheap." Each request owns cache.

Clients are your BFF/API. Stores are optional: prompt cache, model artifacts in object storage. Failure is OOM and thermal throttling as much as network.

A simple example

Your chatbot replica receives "Summarize this ticket." It forwards to inference-server:8000 with a 2k-token prompt. Prefill takes 80 ms, first token streams, decode runs at ~40 tokens/s. Meanwhile the server has 12 other chats in the same decode batch. A 32k-token RAG prompt arrives; KV cache would exceed VRAM. The server rejects with 503 or preempts a low-priority job. Your API maps that to "try a smaller context" or a queue, not a hang.

Calling the GPU from the Django request thread with model.generate() and batch_size=1 leaves the GPU idle between Python round trips. That is why a dedicated server exists.

Common mistakes

One request per GPU with no batching. You pay for an H100 and run it like a laptop.

Ignoring KV cache in capacity plans. Concurrent users × context length is the real limit.

Reloading the model per request or mixing many models on one GPU without a scheduler (post 77).

Treating the inference server as infinitely horizontally scalable without model size: 70B may be 8 GPUs per replica.

No timeout / max_tokens. A runaway decode holds the batch slot forever.

How this shows up in real systems

vLLM, Hugging Face TGI, TensorRT-LLM, NVIDIA Triton, llama.cpp/server: the common serving stacks.
OpenAI, Anthropic, Bedrock, Vertex, Together: you skip owning this box; you still design timeouts, retries, and streaming on your side.
Internal "model gateways": auth, routing, and billing in front of many servers.

The user-visible trick is not waiting for the full answer. That is streaming.

Recap

An inference server is a long-lived GPU process with weights loaded: prefill, KV cache, decode, batch.
Capacity is VRAM and tokens/s, not QPS in the web sense.
Product APIs call this server; they should not own the CUDA loop.

Users hate spinners that last ten seconds. Next: stream the tokens out.

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 11 of 26

← Previous: Designing a File Storage System → Next: Streaming LLM Responses