Model Routing

Pradeep P
3 days ago
4 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 16 of 26

← Previous: GPU Scheduling → Next: AI Rate Limiting

Layer 6 — Modern systems · Post 78 of 88

Model routing picks which model should handle a request based on cost, quality, latency, or task type — a load balancer for intelligence.

What you'll learn

Why "always call the smartest model" is a cost and latency bug, not a strategy
Heuristics, classifiers, and cascades — and when to fall back
How routing keys (task, tenant, region) belong in the same design as GPU pools

The idea in one minute

You do not have one model. You have a menu: a cheap 8B for classification, a mid-size for chat, a frontier model for hard reasoning, a specialized code model, an embedder.

Model routing is the layer that maps a request to a model (and often a provider) before you spend tokens:

Request | v [ Router: rules / classifier / cascade ] | | | v v v small default frontier chat chat (escalation)

It is a load balancer whose backend is quality × dollars × ms, not just CPU.

Why it matters

Frontier APIs can be 10–50× the price of a small model for work a small model already does well: "is this spam?", "rewrite in a friendly tone," "extract the date." Sending everything to the largest model will:

Blow the bill (post 88).
Blow latency (big models, long queues).
Blow capacity (your GPU pool for the 70B is tiny).

In interviews, "we'll use GPT-4 for everything" is a smell. The follow-up is how you choose, how you measure quality, and what happens when the primary is rate-limited.

How it works

What you route on

Task type: embed vs generate vs rerank vs vision. Different binaries, not just sizes.
Difficulty: short FAQ vs multi-step analysis. Often a cheap model classifies first.
SLO: autocomplete needs 100 ms; a report can wait 20 s.
Tenant / tier: free users get the small model; enterprise gets the large one, or a higher cap.
Data gravity / region: EU traffic stays on an EU endpoint; medical text stays on a HIPAA-capable vendor.
Health: if provider A is 429ing, fail over to B or to a smaller local model.

Three common designs

Rules. if path == /embed → embedder. if max_tokens < 64 and prompt < 500 → 8B. Boring. Debuggable. Start here.

Classifier / semantic router. Embed the prompt or run a tiny model: simple | coding | reasoning. Then pick. You pay a small extra hop to save a large one. Tune the labels; silent misroutes are worse than a high bill you can see.

Cascade. Try the cheap model. If a confidence or verifier says "weak," retry on the expensive model. Great for support bots. Dangerous if the cheap model is confidently wrong — you need a real signal (structured parse fail, retrieval miss, user-visible uncertainty), not vibes.

Keep the contract stable

The client should not know the model name if you can avoid it. Expose capabilities ("chat", "strong-reason", "json-mode"). Pin a model for a prompt version in experiments so evals stay comparable. Log model_id on every trace (post 87).

Routing must include prompt + tools + temperature, or you "route" to a model that cannot call the tools the product needs.

A simple example

A helpdesk copilot:

Classify intent with a 1B model: reset_password | billing | other.
reset_password hits retrieval + a small model with a tight template. Cost: fractions of a cent.
billing goes to a mid model with RAG over invoices.
other, or any answer the reranker scores poorly, escalates to the frontier model once.

A spike in "how do I reset my password?" never touches the expensive queue. A gnarly refund thread still can. When Anthropic 429s, billing falls back to your hosted 70B with a banner that answers may be slower.

Common mistakes

Routing only on user id. Power users still ask easy questions. Route on task, then apply a tenant cap.

No fallback. One vendor outage becomes a site outage. Have a degraded model, even if quality drops.

Cascade without a stop. Cheap model fails, expensive model fails, you loop. Cap retries. Surface "I don't know."

Hiding the model from observability. You cannot A/B or debug cost if every log says assistant.

Changing the model under an eval. Quality "regressed" because routing shifted. Freeze the route in benchmarks.

How this shows up in real systems

OpenRouter, Helicone, Portkey, LiteLLM: proxies that pick providers by cost, latency, or fallbacks.
Bedrock / Vertex / Azure: model IDs in one API; you still choose.
Internal "model gateway": the same pattern as an API gateway, with token accounting.
MoE inside a model is not routing. Your product still routes between models and tools.

The GPU scheduler (previous post) packs work onto silicon. The router decides which silicon and which weights that work is allowed to use.

Recap

Routing is a policy: cost, quality, latency, region, and health — not a single default model.
Prefer rules, then a tiny classifier, then a bounded cascade.
Keep a stable product contract, log the real model_id, and always have a fallback.