Caching

Pradeep P
3 days ago
4 min read

Series: Modern System Design · Layer 1 — The building blocks

Layer 1 · Post 9 of 15

← Previous: CDNs → Next: Redis

Layer 1 — The building blocks · Post 9 of 88

Caching stores the answer to an expensive question so the next request can skip the hard work. The hard part is knowing when that answer is stale.

What you'll learn

Why caches exist at many layers, not just Redis
Hit, miss, TTL, invalidation, and stampede in one picture
The rule: a cache is a performance optimization, not the source of truth

The idea in one minute

Some work is slow or expensive: a disk read, a join across big tables, a call to another team, a GPU inference.

A cache keeps a recent result in a faster place (memory, edge, browser). The next time you see the same question, you return the saved answer.

That saved answer can be wrong the moment the real data changes. Caching is the art of being a little wrong on purpose, in ways users can live with — or of being fast at throwing the old answer away.

Why it matters

You cannot scale a database by asking it the same popular question a hundred thousand times a second. You scale by not asking.

Caches:

Drop latency (RAM vs disk vs another continent).
Drop load on the source of truth.
Smooth spikes (a celebrity logs in; their profile is fetched once, served many times).

They also create the famous bugs: "I updated it but the app still shows the old value" and "the cache expired and we DDoSed ourselves."

How it works

Layers (you already have several)

Layer: Browser; Example: HTTP cache, localStorage; Typical lifetime: Seconds to days
Layer: CDN; Example: Edge cache; Typical lifetime: Seconds to a year (immutable files)
Layer: App in-process; Example: Local hashmap; Typical lifetime: Process lifetime, one machine only
Layer: Shared cache; Example: Redis / Memcached; Typical lifetime: Seconds to hours
Layer: Database; Example: Buffer pool, query cache; Typical lifetime: Internals you barely configure

A request might hit three of these. That is good for speed and bad for "where is the data?"

The basic loop

request
   → look up key
        hit  → return value
        miss → compute / fetch from DB
             → store key → value (with TTL)
             → return value

Key must match the question: user:42:profile is not user:42:settings. Bad keys mix data. Too-specific keys never hit.

TTL (time to live) is the lazy way to become correct again: "forget this after 60 seconds." Simple. You live with up to 60 seconds of staleness.

Invalidation is the eager way: when the profile is saved, delete user:42:profile. Correct sooner. Easy to forget a key, or to race (delete, then a stale writer fills the cache again).

Stampede (thundering herd)

A hot key expires. Ten thousand requests miss at once and all hit the database. The cache was protecting you; the expiry synchronized the herd.

Mitigations: slightly random TTLs, single-flight (only one request refills; others wait), soft TTL (serve stale while one refresh runs).

Aside vs write-through vs write-around

Cache-aside (lazy): app reads cache; on miss, reads DB, fills cache. Most web apps.
Write-through: writes go to cache and DB together. Cache stays warmer; writes get slower.
Write-around: writes skip the cache; next read fills it. Avoids filling the cache with one-time writes.

A simple example

Product page GET /products/9. The DB query takes 15 ms. At 50k QPS that is a lot of 15 ms.

You cache product:9 in Redis for 30 seconds. p50 becomes ~1 ms. A price change can be wrong for up to 30 seconds. For a bookstore, fine. For a flash sale, you shorten TTL, invalidate on price update, or skip the cache on the "buy" path.

The inventory count on the same page might not be cached, or cached for 2 seconds, because overselling is costly. Same page, two policies. That is normal.

Common mistakes

Using the cache as a database. Restart Redis and your only copy of orders is gone. Source of truth stays on disk (or a real DB).

Caching errors. A 500 from a dependency gets stored for 10 minutes. Now everyone gets a fast 500.

Unbounded keys. Caching per ?page= with infinite crawlable URLs fills memory with junk. Cap cardinality.

Tiny TTLs "to be safe" on huge objects. You pay serialization cost constantly and still stampede.

Forgetting privacy. Caching Authorization-specific JSON under a URL-only key shares one user's data with another.

How this shows up in real systems

Facebook/Twitter-style feeds: caches for timelines, social graphs, sessions.
E-commerce: CDN for assets, Redis for sessions and hot products, DB for checkout.
DNS itself (previous post) is a cache. So is your CPU's L1. The idea repeats at every scale.

The next post is the shared cache most backend teams actually run: Redis.

Recap

A cache stores expensive answers closer and cheaper.
Staleness vs load is the tradeoff. TTL and invalidation are the knobs.
Protect the origin from stampedes, and never let the cache be the only copy of something you cannot lose.