Token Streaming

Pradeep P
3 days ago
4 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 13 of 26

← Previous: Streaming LLM Responses → Next: LLM Request Queues

Layer 6 — Modern systems · Post 75 of 88

Token streaming is the protocol-level idea: the model emits pieces of text over time, and every layer from GPU to browser has to stay in sync with that flow.

What you'll learn

What a token is (and why it is not always a word) and how detokenization hits the stream
How SSE deltas, usage counters, and [DONE] line up with decode steps
Why UTF-8, tool-call JSON, and "partial tokens" break naive string concat in the UI

The idea in one minute

The model does not emit "English." It emits token IDs from a tokenizer (BPE/unigram). The inference server detokenizes incrementally and sends deltas — usually a small string — on each decode step (or every few steps). Every hop must preserve order, not drop a chunk, and not assume each chunk is a complete word or a valid UTF-8 character.

GPU decode --> token id | v [ Detokenizer: id --> incremental text ] | v [ SSE: data: {"choices":[{"delta":{"content":"Hel"}}]} ] | v [ Your API: pass-through / add request_id ] | v [ Client: append delta, render markdown cautiously ] | missing chunk? garbled words; double-apply? duplicated letters

Post 74 was "why stream." This is the wire format and the token boundary problem.

Why it matters

If you buffer until a newline, you destroy TTFT for tokens that are sub-word pieces. If you JSON.parse a tool call from incomplete deltas, you throw in the UI. If you count "words" for billing, you will disagree with the vendor's token invoice.

Interviews that go deep on LLM infra want you to separate tokens (model/billing) from characters (UI).

How it works

Tokenizer. Hello might be one token; Supercalifragilistic might be several. Numbers and code tokenize differently from prose. max_tokens is this vocabulary, not whitespace words.
Decode step. The GPU produces the next id given the KV cache. The server maps id → string piece. Some pieces are leading spaces (" world"). Some are incomplete UTF-8 until the next token arrives — a careful detokenizer holds a tail so you do not emit a broken character.
Wire protocol. OpenAI-compatible streams: each event is data: \n\n. delta.content is a string fragment. Later, finish_reason (stop, length, tool_calls). data: [DONE]\n\n ends the stream. Usage (prompt_tokens, completion_tokens) may arrive at the end only — your live UI cannot show exact cost until then unless you count locally with the same tokenizer.
Your API. Prefer pass-through of frames. If you coalesce (send every 50 ms), you trade smoothness for fewer syscalls — say so. Do not reorder. Do not UTF-8-split a delta in the middle of a codepoint when chopping for a max chunk size.
Client. Maintain fullText += delta. Render markdown on a debounce so a half-open **bold does not flicker. For tool calls, accumulate delta.tool_calls[].function.arguments as a string until finish_reason; then JSON.parse.
Failure. A dropped SSE event (rare on HTTP/2, possible with bad proxies) desyncs the text. There is usually no per-token replay. You reconnect with a new request or show "interrupted." Duplicate delivery of one event doubles characters — parsers must be dumb append, not "smart merge."

Clients are the browser parser. APIs are frame-preserving proxies. Stores might save the final concatenated string plus token counts. Failure is framing and partial JSON.

A simple example

The model emits ids that detokenize as "Hel", "lo", "!". Three SSE events; the UI shows Hello!. A later token is the start of an emoji that needs the next id to become a valid character; the server holds it for one step, then emits "🎉". Your naive server that flushed raw UTF-8 bytes on a 1-byte buffer would have sent a replacement character.

A tool call streams {"name":"get_weat then her","city":"Pune"}. You do not parse until the stream finishes that tool block. Billing uses 14 completion tokens, not 3 "words."

Common mistakes

Treating each SSE event as a word or a sentence. Tokens are subword. UI must append blindly.

Parsing JSON tool arguments on every delta. It will throw until the last chunk.

Re-encoding or logging with the wrong tokenizer and then arguing with finance about token counts.

Splitting chunks on byte length and breaking UTF-8 or combining characters.

Assuming usage is in the first event. Often it is only at the end; design the cost UI for that.

How this shows up in real systems

OpenAI, Azure OpenAI, Grok, Anthropic streaming APIs: delta events; details differ, the append model does not.
vLLM / TGI: --return-tokens vs text deltas; same decode loop underneath.
tiktoken, Hugging Face tokenizers: the local way to estimate what the server will bill.

Tokens on the wire still need a queue when GPUs are busy. That is the next post.

Recap

Streaming is ordered token-id → detokenize → delta frames.
The UI appends; it does not wait for word boundaries. Tool JSON waits for completion.
UTF-8, dropped frames, and usage-at-end are the sharp edges.

When more requests arrive than decode slots, you queue — next in the series.

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 13 of 26

← Previous: Streaming LLM Responses → Next: LLM Request Queues