Semantic Caching

Pradeep P
3 days ago
4 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 18 of 26

← Previous: AI Rate Limiting → Next: Vector Databases

Layer 6 — Modern systems · Post 80 of 88

Semantic caching reuses a previous LLM answer when a new question is close enough in meaning, which can cut both latency and spend.

What you'll learn

Exact-match cache vs cache-by-meaning, and when each is safe
How embeddings, a similarity threshold, and a cache key actually work
Why personalized, time-sensitive, or tool-backed answers will bite you if you cache them

The idea in one minute

Layer 1 caching: same key, same bytes. LLM traffic is messier. Users type "hours of the Berlin store" and "when does the Berlin shop open?" — two strings, one intent.

Semantic caching stores past (query → answer) pairs, embeds the new query, and if a neighbor is close enough, returns the stored answer without calling the model:

New question | v embed → nearest neighbor in cache | +-- similarity >= threshold → return cached answer +-- else → call LLM → store (embedding, answer, metadata)

You trade a small embedding lookup for a large generation. You also trade freshness and correctness if you pick a sloppy threshold.

Why it matters

Generation is slow and priced per token. Support bots, docs Q&A, and "explain this error" flows repeat. A 30–70% cache hit rate is a real cost and latency lever — often bigger than shaving a few tokens off the prompt.

Interviewers will ask how this differs from Redis GET prompt. The answer is approximate matching. Then they will ask when you must not do it. Personalized, stateful, and ticking-clock answers are the trap.

How it works

Exact cache still comes first

Hash model + temperature + system prompt + tools + normalized user text. If someone retries the same string, skip embedding. Cheap. Exact.

Normalize lightly: trim whitespace, maybe lowercase. Do not "normalize away" numbers and names (order 1842 vs order 1843).

Semantic lookup

Embed the query with a stable embedding model (changing it invalidates the space).
k-NN search over cached queries (this is why vector DBs are next).
If cosine_similarity >= τ (you tune τ, often 0.90–0.97), treat as a hit.
Optionally confirm with a tiny classifier: "same question?" False friends are common (reset password vs reset MFA).

Store answers, not just embeddings. Attach: model id, prompt version, tenant, language, retrieved doc ids. A hit is only valid if those match. Caching "what is our refund policy?" across tenants is how you leak.

Invalidation

Docs change. Prices change. A semantic cache without TTL is a rumor mill.

TTL by domain (FAQ: hours; weather: minutes; "my balance": never).
Purge by doc_id when an article updates. If the answer depended on RAG, put a content hash in the key or you serve a correct-looking stale paragraph.

Where it sits

In front of the LLM, after auth and routing, before generation. Do not cache across users unless the content is public. Do not cache tool results that have side effects.

A simple example

Your docs bot gets "How do I rotate API keys?" a thousand times a day, plus "steps to rotate an API key?" and "API key rotation?"

First ask: miss, generate, store. The paraphrases hit at τ=0.93. Time-to-first-token drops from 800 ms to 40 ms. Cost drops with it.

Then you ship a new key-rotation UI. If you only TTL 24 h, people follow the old clicks for a day. You purge cache entries tagged docs:api-keys on publish. Hits resume after the next miss.

A user asks "what's in my last invoice?" Similarity to someone else's invoice question is high. You do not semantic-cache: the key includes user_id and you skip semantic match for any prompt that includes private tools.

Common mistakes

Threshold too low. You answer "refunds in 30 days" to "refunds in 14 days." High τ, plus exact cache for the rest.

Caching completions with high temperature / creative writing. Similar questions should not share a poem.

Putting PII in the cache without isolation. Treat it like a database of past user text. Encrypt, tenant-partition, TTL.

Forgetting the rest of the prompt. Same question, different system prompt or model = different answer. Put them in the key.

No metrics. Hit rate, false-hit reports (thumbs down after a cache hit), and latency of the embedding path. A slow embedder can erase the win.

How this shows up in real systems

GPTCache, Redis + vector, Postgres pgvector: common DIY.
Helicone / Cloudflare AI / vendor "prompt caches": some are prefix caches (provider KV for repeated system prompts), which is not the same as semantic Q&A reuse. Both save money; do not confuse them in an interview.
CDN analogy: semantic cache is a CDN for meaning, with a worse invalidation story.

Rate limits (previous) stop spend. This layer avoids spend. To look up "near enough," you need a place to store vectors.

Recap

Exact-match first; semantic match for paraphrases with a high threshold and a strict key (model, prompt, tenant).
Invalidate on knowledge change; never semantic-cache private or real-time answers.
Measure false hits, not only hit rate.

The lookup engine for that "near enough" is a vector database.

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 18 of 26

← Previous: AI Rate Limiting → Next: Vector Databases