Streaming LLM Responses

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 12 of 26

← Previous: What Is an LLM Inference Server? → Next: Token Streaming

Layer 6 — Modern systems · Post 74 of 88

Streaming sends tokens to the client as they are produced so users see an answer forming instead of staring at a spinner.

What you'll learn

Why time-to-first-token beats "total generation time" for perceived performance
How SSE (and sometimes WebSockets) carry chunks from the inference server through your API to the browser
What cancellation, proxies, and buffering do to a stream in production

The idea in one minute

Non-streaming generation waits until max_tokens or a stop sequence, then returns one JSON blob. Streaming returns pieces as decode produces them. The UI types. Perceived wait is time to first token (TTFT), not the full 8-second answer.

Browser / mobile | | fetch() + ReadableStream (SSE) v [ Your API gateway ] ---- must not buffer the whole body ---- | v [ Inference server stream ] | client disconnect? cancel GPU work (best effort) nginx buffers? user sees nothing until flush — disable proxy buffering

This post is the product and HTTP design. The next post is the token/protocol mechanics inside that stream.

Why it matters

A 200 ms TTFT and a 20 token/s drip feels alive. The same job as a 6 s spinner feels broken — even if total time is identical. Interviewers for chat UIs expect streaming, cancellation (user hits Stop), and "we did not buffer in nginx."

Retries are awkward: you cannot blindly replay a stream without duplicating text in the UI unless you design it.

How it works

Client opens POST /v1/chat/completions with stream: true (or Accept: text/event-stream). It reads an incremental body. WebSockets are used when you already have a bidirectional channel (chat apps); SSE is enough for model output and is simpler through HTTP/2.
Your API authenticates, applies quota, maybe RAG, then proxies the stream. It should forward chunks as they arrive. If you must log the final answer, append in memory (with a size cap) or assemble in a worker after the stream ends — do not wait to start sending.
Inference server emits chunks (OpenAI-style data: {delta} then data: [DONE]). Your API may rewrite errors into a final SSE event so the client has a single parser.
Cancellation. User clicks Stop: the browser abort()s. Your API should close the upstream and, if the server supports it, cancel the job so the GPU slot frees. If you ignore abort, you pay for tokens nobody will see.
Failure. Mid-stream GPU blip: send an error event and close. The UI keeps tokens already rendered and offers Retry (new request, not a resume unless you built one). Idle timeouts on load balancers (60s) will kill long answers — raise them or send heartbeats/comments in SSE.
Backpressure. If the client is slow (mobile), TCP windows fill. A well-behaved server slows decode or drops the request. Unbounded server-side queues of unsent tokens waste memory.

Stores: optionally persist the conversation after DONE. Failure: proxies that buffer, and gateways that wait for Content-Length.

A simple example

The user asks for a trip plan. TTFT is 300 ms; the first words appear. At 4 s they hit Stop. AbortController fires; your API closes the vLLM request; the batch slot is reused. You save a partial assistant message with status=cancelled.

If API Gateway was set to buffer until EOS, the user would have stared at a blank bubble for the whole generation. You set X-Accel-Buffering: no (or cloud equivalent) and chunked transfer.

Common mistakes

Collecting the full completion in the BFF, then streaming from memory. That is fake streaming. TTFT includes the whole model time.

Default nginx proxy_buffering on. Classic "works on localhost, dead in prod."

No abort path. Stop button is cosmetic; the GPU keeps running.

Using WebSockets only because "it's realtime" when SSE would do. Extra stateful gateway for little gain.

Retrying a failed stream by concatenating a second completion without a delimiter. The UI doubles paragraphs.

How this shows up in real systems

ChatGPT, Claude.ai, Gemini, Cursor: SSE or similar chunked HTTP to the client.
OpenAI stream=true, Anthropic SSE: the vendor contract you often wrap.
Vercel AI SDK, LangChain callbacks: client helpers; you still own proxy buffering and cancel.

Streaming is the UX. Token streaming is how the bytes are chunked and aligned with the model.