RAG Architecture
Series: Modern System Design · Layer 6 — Modern systems
Layer 6 · Post 20 of 26
← Previous: Vector Databases → Next: Building an AI Agent
Layer 6 — Modern systems · Post 82 of 88
RAG retrieves relevant documents at query time and feeds them to the model so answers can be grounded in your data instead of only in training weights.
What you'll learn
The ingest vs query paths: chunk, embed, retrieve, (rerank), generate
Why grounding is a system design, not "paste the wiki into the prompt"
Failure modes: wrong chunks, bloated context, and answers that ignore the evidence
The idea in one minute
Weights freeze at train time. Your refund policy changed Tuesday. Retrieval-Augmented Generation (RAG) pulls relevant snippets at query time and puts them in the prompt so the model can answer from your text.
Ingest: docs → chunk → embed → vector index (+ keyword index) Query: question → retrieve k chunks → rerank → prompt + generate ↘ cite sources
The model is a reasoner over a working set, not a database. If retrieval misses, generation will improvise. That improvisation is called a hallucination when it looks confident.
Why it matters
Fine-tuning for every doc change is slow and expensive. Stuffing the entire corpus into context is impossible (and pricey). RAG is the scalable compromise: search, then read.
Interviewers use RAG to test whether you remember Layer 1 (cache, indexes) and Layer 2 (search). They will ask how you keep answers fresh, permissioned, and citable. "We use LangChain" is not an architecture.
How it works
Ingest (offline / async)
Connectors: wiki, PDFs, tickets, code. Respect ACLs from day one.
Parse to text. Tables and scans fail here more than in the vector DB.
Chunk with overlap. Too small: no context. Too big: noisy retrieval and token burn. Start near 300–800 tokens and measure.
Embed and upsert. Store doc_id, version, ACL, checksum.
Re-index on change. Treat ingest as a pipeline with lag SLOs ("policy live within 15 minutes").
Query (online)
Optional query rewrite / HyDE: expand "it failed" into a searchable question.
Hybrid retrieve: BM25 + vectors, filtered by tenant and permissions.
Rerank a larger candidate set (cross-encoder). This step often beats a fancier vector DB.
Budget the prompt: top chunks that fit, maybe compress. Leave room for the question and the answer.
Instruct the model: answer only from the snippets; say you don't know; cite chunk ids.
Log retrieved ids with the trace. You cannot debug "wrong answer" without knowing what it read.
Grounding is a contract
You still need: refusal when snippets are irrelevant, and a UI that shows sources. RAG without citations is a chatbot with extra latency.
Multi-hop questions ("compare last year's policy to this year's") may need multiple retrievals or an agent loop (next posts). One-shot RAG is for one-hop "what's in the doc?"
A simple example
Employee asks, "Can I expense a second monitor?"
Retrieve: expense policy chunk, hardware stipend chunk, a stale Slack rumor (filtered out — not in the corpus, good). Rerank keeps the two policy chunks. Prompt says quote the stipend cap. Model answers "$250 once per 24 months" and cites Policy §4.
Without retrieval, a general model invents a number. With retrieval but no ACL filter, an intern might retrieve exec-only compensation docs. The architecture includes authz on retrieve, not only on the HTTP route.
You update the stipend to $400. Ingest job re-chunks that page. Until then, RAG is correctly wrong. Freshness is an ingest SLO, not a model parameter.
Common mistakes
Dumping k=20 huge chunks. The answer is in chunk 2; the model attends to a FAQ in chunk 17. Smaller k after rerank, or hierarchical retrieve (doc then section).
No eval set. You need questions with labeled passages, not only "the bot seemed nice." Track retrieval recall and answer faithfulness separately.
Embedding the query with a different model than ingest.
Refreshing the index weekly for a product that changes daily.
RAG for math, live stock, or "delete this user." Those need tools (post 85), not a wiki snippet.
Ignoring prompt injection in retrieved content ("ignore previous instructions"). Treat retrieved text as untrusted.
How this shows up in real systems
Internal copilot / customer support: the textbook RAG app.
Managed: Amazon Kendra + Bedrock, Azure AI Search, Vertex Search, Glean-style products.
DIY: parsers → chunker → pgvector / Pinecone → Cohere rerank → your inference server.
Long-context models reduce chunk pain; they do not remove permissions, citations, or ingest.
Vector DBs (previous) are the index. RAG is how you use them in the request path without lying about what the model knows.
Recap
RAG is retrieve then generate, with ingest freshness and ACLs as first-class parts.
Hybrid search + rerank + a token budget beats "more chunks."
Cite sources, eval retrieval and answers separately, treat retrieved text as untrusted.
When the task takes several retrieves and actions, you are no longer in one-shot RAG. You are building an agent.
Series: Modern System Design · Layer 6 — Modern systems
Layer 6 · Post 20 of 26
← Previous: Vector Databases → Next: Building an AI Agent
Comments