Cache Stampede

Pradeep P
2 days ago
3 min read

Series: Modern System Design · Layer 1 — The building blocks

Layer 1 · Post 20 of 21

← Previous: Consistent Hashing → Next: Hot Keys and Hot Partitions

Layer 1 — The building blocks · Post 93 of 119

When a hot cache entry expires, every client can miss together and crush the source of truth.

What you'll learn

Why a perfectly healthy TTL can still take down the database
Locks, early refresh, stale-while-revalidate, and jitter
How this differs from a hot key that never expires

The idea in one minute

A cache stampede (also dogpile, or thundering herd on a key) is: many clients need the same value, the cache is empty for that key, and they all hit origin together.

The usual trigger is TTL expiry of a popular key. One millisecond the homepage is served from Redis. The next, 10,000 app threads miss, run the same expensive query, and the primary CPU goes to 100%.

t < TTL:   clients --> cache (hit) --> done
t = TTL:   clients --> cache MISS --> origin  }  all at once
           clients --> cache MISS --> origin  }

Post 9 said a cache is not the source of truth. A stampede is when the cache being empty synchronizes everyone onto that truth.

Why it matters

This is a named outage mode, not a trivia name. It shows up after deploys (empty cache), after failovers, after % N remaps (previous post), and every night when a "daily" key expires at midnight UTC.

Interviews: they want jittered TTLs and single-flight (only one refresh), not "we'll add more Redis." More Redis does not help if every replica misses the same key.

How it works

Jitter the TTL. Do not expire homepage at exactly 60s for every writer. Use 60s ± 10%. Related keys should not share a clock.

Single-flight / mutex. On miss, one caller fetches origin; others wait on that result (or serve stale). Redis SET key NX as a lock, singleflight in Go, request coalescing in a local process. The lock must expire so a crashed refresher does not brick the key.

Stale-while-revalidate. Keep serving the old value past TTL while a background job fills the new one. Users see slightly stale; origin sees one query. HTTP stale-while-revalidate is this at the CDN.

Probabilistic early expiration (XFetch / "expire slightly early under load"): as TTL nears, some fraction of requests refresh before empty. Load spreads.

Warm on deploy. Empty caches after a bounce are a stampede you scheduled. Pre-fill hot keys or use a cache that survives process restart (Redis vs in-process).

Do not use TTL=0 "to be safe" on the hottest keys if origin cannot take the QPS. Pin them and invalidate on write instead.

A simple example

A product page caches GET /products/42 for 30 seconds. 8,000 QPS on that SKU during a sale. At second 30, 8,000 misses. The query is 50 ms with a join. Little's Law: a sudden extra 8,000 × 0.05 ≈ 400 extra in-flight queries. The pool is 100. Timeouts cascade.

Fix: one locker refreshes; everyone else waits 50 ms for Redis to fill; or serve the 30-second-old HTML for another 2 seconds while refresh happens. Origin sees one query, not 8,000.

Common mistakes

TTL aligned to cron. "Expire all listing pages at 0 * * * *" is a calendar stampede.

Lock without TTL. Refresher dies; every request waits on a lock that never drops.

Only process-local singleflight. 200 pods × 1 refresh = 200 origin queries. You need a distributed lock or a shared cache that already has the new value from the first winner.

Calling every overload a stampede. A hot key that is always in cache can still saturate one shard (next post). Stampede is specifically synchronized misses.

How this shows up in real systems

CDN origin shield / request collapsing: stampede protection at the edge.
Nginx proxy_cache_lock, Varnish grace: one miss fills, others wait or serve stale.
Memcached/Redis plus app-level locks: DIY; easy to get wrong, still common.
Kubernetes rolling deploys: in-process caches empty pod by pod — usually fine; a fleet restart is not.

Recap

Stampede = many misses at once on a hot key, usually at TTL.
Jitter, single-flight, stale-while-revalidate beat a bigger origin.
Next: when one key is hot even when the cache is full — hot keys and hot partitions.