Rate Limiting

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 3 — Reliability

Layer 3 · Post 7 of 14

← Previous: Bulkheads → Next: Backpressure

Layer 3 — Reliability · Post 32 of 88

Rate limiting caps how often a client can call you. It protects shared resources from abuse, bugs, and sudden popularity.

What you'll learn

Token bucket vs sliding window — what you actually enforce
Where to put the limiter (edge, gateway, service) and what key you count on
How 429 and Retry-After talk to well-behaved clients (and what to do with the rest)

The idea in one minute

A rate limit is a policy: this client may do N operations per window (or a sustained rate with bursts).

API key "acct_9" 100 requests / minute request 101 --> 429 Too Many Requests Retry-After: 12

Bulkheads cap your concurrency. Rate limits cap their arrival rate before they fill those bulkheads. A buggy loop, a scraper, or a famous launch should not be indistinguishable from a DDoS on your Postgres.

Why it matters

Without a limit, capacity planning is fiction. One tenant with a tight while true loop is a reliability incident. Interviews almost always want a limit on public APIs: per IP, per user, per API key, and a global ceiling.

It is also product: GitHub and Stripe publish quotas. That is how you stay multi-tenant without one customer eating the fleet.

How it works

Common algorithms (no need for a table — pick one and know the failure mode):

Fixed window: 100/minute, counter resets at :00. Cheap. Burst of 200 at the window edge (100 at 00:59, 100 at 01:00).
Sliding window / sliding log: smoother, more state (Redis sorted sets, or approximate sliding counters).
Token bucket: tokens refill at rate r, bucket size b allows bursts. Good default for APIs. Leaky bucket is the dual: smooths outflow.

Store the counter in Redis or at the edge (Cloudflare, API Gateway, Envoy local rate limit). In-process counters do not work across 20 pods unless you shard keys (which you still should: key → limiter instance).

What you key on: IP (coarse, NAT hides many users), user id, API key, route (POST /charge stricter than GET /health). Often all of them: a global cap plus a per-key cap.

HTTP: 429, Retry-After, and headers like X-RateLimit-Remaining so official SDKs can back off (Post 28) instead of retrying immediately.

Rate limiting is policy. Backpressure (next) is "I am actually full right now." You want both: steady quotas, plus emergency shedding when queues blow up.

A simple example

You ship a public search API. Fair use: 10 qps per API key, burst 20.

A partner's cron jobs fire 200 qps after a deploy. Redis token bucket on the key returns 429 in the API gateway (Envoy or AWS API Gateway) before the query hits Elasticsearch. Partner SDK respects Retry-After. Your p99 for everyone else stays flat.

A second key is a scraper rotating IPs. Per-key is not enough; you add per-IP and a WAF at Cloudflare. Different keys, same idea: bound arrival.

Common mistakes

Limit only at the app. They can still crush the gateway, TLS, and your mesh. Enforce at the edge for abuse; enforce in the app for business quotas.

One global limit, no per-tenant. The biggest customer is always at the ceiling; everyone else starves — or the opposite: one abuser eats the global cap.

Silent drop. Prefer 429 for APIs so clients can back off. Drops are for volumetric attacks at the CDN.

Retrying 429 immediately. You taught the client to hammer. Use backoff and Retry-After.

Limits that ignore expensive vs cheap routes. GET /health and POST /export.csv are not the same unit. Cost-based or per-route limits.

How this shows up in real systems

GitHub, Stripe, Twitter/X, Twilio: documented REST quotas, 429s, SDK backoff.
AWS API Gateway, Cloudflare, Fastly, Envoy: token bucket / local+global rate limits.
Redis + INCR/EXPIRE or token-bucket scripts: the DIY that every backend eventually writes.
nginx limit_req: still everywhere.

Recap

Rate limits cap arrival rate per key so abuse and bugs cannot consume the system.
Token bucket (burst + refill) is the usual API choice; return 429 + Retry-After.
Policy at the door. Next: backpressure when you are full anyway.