AI Rate Limiting

Pradeep P

Aug 31

4 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 17 of 26

← Previous: Model Routing → Next: Semantic Caching

Layer 6 — Modern systems · Post 79 of 88

AI rate limiting is not just requests per second. Tokens, GPU-seconds, and cost per user all have to be capped or someone will melt the bill.

What you'll learn

Why RPS limits fail for LLMs and what you cap instead (tokens, concurrency, dollars)
Token buckets that spend tokens, plus in-flight limits for KV cache
How to talk about 429s, retry-after, and tenant fairness in an interview

The idea in one minute

Classic APIs: 100 requests/second per key. LLM APIs: one "request" can be 8 tokens or 80,000, and it can hold a GPU for a minute.

AI rate limiting meters the scarce things:

Request → auth / tenant → check: RPM, TPM (tokens/min), concurrent gens, $ budget → allow | delay | 429

If you only count HTTP calls, a user with a 128k paste is "one request" and a user saying "hi" is also "one request." Your fleet disagrees.

Why it matters

Without this, you get:

Bill shock on hosted APIs (one intern loop, one public key in a frontend).
Noisy-neighbor on self-hosted GPUs (one tenant's agent loop occupies every slot).
Retry storms that turn a 429 into a self-DDoS.

Interviewers expect you to know OpenAI-style RPM vs TPM, and that concurrency is a separate knob because of memory, not because you like extra YAML.

This is Layer 1 rate limiting with a different unit of work. The unit is tokens and GPU-seconds, not requests.

How it works

Dimensions that actually bite

Requests per minute (RPM): stops chatty clients and simple abuse. Necessary, not sufficient.
Tokens per minute (TPM): input, output, or both. Output is harder: you only know the cap (max_tokens) up front; actual length is later. Reserve max_tokens on admit, refund the unused, or you over-admit.
Concurrency: max live generations per key / tenant / global. This is how you protect KV cache.
Cost budget: dollars per day per tenant, with a hard stop, not only a dashboard.
GPU-seconds (self-hosted): same idea as cost, closer to the metal.

Token bucket, but for tokens

A bucket refills at tokens_per_minute / 60. Each request tries to spend estimated prompt tokens + reserved output. If the bucket is empty, 429. Bursts are allowed up to bucket size; sustained rate cannot exceed refill.

Do this per tenant and per API key, and keep a global emergency brake.

Where it lives

Edge / gateway: cheap RPM, API-key checks, bot filters.
Model gateway: TPM, model-specific caps (the 70B pool is smaller than the 8B pool).
Worker: last-resort concurrency; do not rely on it alone.

Spread state (Redis) so many API nodes share the same counters. Local-only limits are how tenant A bypasses you by hitting five pods.

Honest 429s

Return which quota hit (tokens, requests, concurrency) and Retry-After. Clients should backoff. Your own retries toward providers need a retry budget so you do not amplify.

Streaming: if you reserved 4k output tokens and the user stops at 40, credit the remainder or your TPM lies for the rest of the minute.

A simple example

Free tier: 20 RPM, 40k TPM, 2 concurrent chats, $1/day. Pro: 10× those numbers.

A student pastes a book (50k input tokens) with max_tokens=4096. You reserve ~54k. Free TPM is 40k → 429 with "tokens," not a mysterious timeout. They split the doc.

A second tab opens another chat. Concurrency is 2; the third tab waits or 429s. A leaked key in a website would have burned $200 by lunch; the $1/day cap stops it.

Overnight, your eval suite uses a separate key with a high TPM so it does not starve customers — and a budget so a bad loop cannot.

Common mistakes

RPS only. The paste-the-PDF user is legal and fatal.

Not reserving output tokens. Ten "write a novel" calls admit, then you 429 mid-stream or OOM.

Per-pod limits. Horizontal scale silently multiplies quota.

Hiding 429 behind a spinner until the client retries 30 times. Fail clearly.

One global TPM for all models. Embedding is cheap and high QPS; frontier chat is not. Split pools.

Forgetting agents. One user message can become 40 model calls. Limit per session / per agent run, not only per HTTP hit from the browser.

How this shows up in real systems

OpenAI / Anthropic / Gemini: RPM, TPM, sometimes TPD; headers for remaining quota.
API gateways (Kong, Apigee, AWS WAF): good for RPM; you still need token accounting in-app.
Redis token buckets, Stripe-style credits: common DIY.
Kubernetes ResourceQuotas do not understand tokens. Do not pretend they do.

Routing (previous post) picks the model. Rate limits decide whether that tenant may spend it right now. Caching (next) is how you spend less for the same question.