Vector Databases

Pradeep P
3 days ago
4 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 19 of 26

← Previous: Semantic Caching → Next: RAG Architecture

Layer 6 — Modern systems · Post 81 of 88

A vector database stores embeddings and finds nearest neighbors, which is how systems retrieve "similar meaning" instead of exact keywords.

What you'll learn

What an embedding is, and why nearest neighbor is the query, not WHERE text =
ANN indexes (HNSW, IVF) and the recall / latency / memory tradeoff
Why metadata filters, ids, and a real source of truth still matter

The idea in one minute

An embedding is a list of numbers that places a chunk of text (or an image) in a high-dimensional space. Similar meaning → nearby points.

A vector database stores those points plus an id, and answers: here are the k nearest neighbors to this query vector, optionally filtered by metadata.

"reset my API key" | v embedding model | v [ vector index: ANN search ] | v chunk ids → fetch original text from object store / DB

It is an index, not your CMS. The documents still live somewhere you can edit and ACL.

Why it matters

Keyword search misses paraphrases. "rotate credentials" and "API key rotation" may share few tokens. Embeddings catch that, which is the retrieval half of RAG (next post) and of semantic cache (previous).

Interviewers want you to say approximate nearest neighbor (ANN), not "we linear-scan 50 million cosine scores in the request path." They also want you to not replace Postgres with Pinecone for orders and users.

How it works

Write path

Take a chunk of text (chunking policy is a RAG concern; the DB just stores vectors).
Embed with a fixed model and dimension (384, 768, 1536, …). Mixing models in one index is garbage.
Upsert id → vector + metadata (tenant, doc_id, ACL, language, updated_at).
The engine builds / updates an ANN index.

Deletes and updates must be real. Stale vectors with live ids are how you retrieve withdrawn policies.

Read path

Embed the query with the same model. Search top-k. You get ids and scores. Then hydrate: load the actual text. Never prompt the LLM with vectors.

Metadata filters (tenant_id = acme, visibility = public) run as pre-filter, post-filter, or hybrid depending on the product. If you skip them, you have a cross-tenant leak. In an interview, mention ACLs before you mention HNSW.

ANN, not exact k-NN

Exact search is O(n). ANN (HNSW graphs, IVF lists, PQ compression) returns good-enough neighbors fast.

You tune:

Recall: did we get the true nearest chunks?
Latency: p95 of search.
Memory / disk: HNSW loves RAM; PQ shrinks vectors and costs recall.

There is no free lunch. "We use HNSW" is not a design unless you say what recall you measured.

Hybrid search

Vectors are weak on rare tokens: SKUs, error codes, names. Combine BM25 / keyword with vector, then rerank. A vector DB that also does sparse search (or you do sparse in Elasticsearch and fuse) is the usual grown-up setup.

A simple example

You index 200k support articles. Each article is split into ~500-token chunks, embedded, stored with article_id and product=billing.

Query: "Why was I charged twice?" Search top 20 in the billing partition, rerank to 5, pass those texts to the model.

A keyword-only index might miss an article titled "Duplicate invoice FAQ." A vector-only index might retrieve a poetic "charged with energy" blog post. Hybrid + filter product=billing keeps you in the right neighborhood.

You republish the FAQ. You re-embed changed chunks and delete removed ones. You do not "wait for the index to forget."

Common mistakes

One giant unfiltered index. Fast demo. Compliance incident.

Re-embedding with a new model into the old index. Dimensions and geometry change. Rebuild.

Tiny or huge chunks stored as 'the document.' Retrieval quality is mostly chunking, not the logo on the vector DB.

Treating scores as probabilities. Cosine 0.81 is not "81% true." Use it to rank, then rerank or let the LLM cite.

No source of truth. If the vector DB is the only copy of the article, you will regret it. Store text in object storage / CMS; the index is derived.

Ignoring ops. Backups, rebuild time, replica lag, and "search while indexing" matter at 100M vectors.

How this shows up in real systems

Pinecone, Weaviate, Milvus, Qdrant, Vespa: dedicated engines.
pgvector, Elasticsearch k-NN, OpenSearch, Redis vector: good when you already live there and scale is moderate.
FAISS / ScaNN: libraries you embed in a service; you own the cluster story.

Semantic cache uses a small vector index of questions. RAG uses a large index of your corpus. Same data structure, different lifetime and invalidation.

Recap

A vector DB is an ANN index over embeddings, plus ids and metadata — not a replacement for your database of record.
Same embedding model in and out; filter by ACL; hydrate text after search.
Trade recall, latency, and memory on purpose; hybrid search saves you on identifiers and rare terms.

Retrieval is a component. RAG is the architecture that feeds those chunks to a model without pretending the weights know your wiki.

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 19 of 26

← Previous: Semantic Caching → Next: RAG Architecture