Agent Memory

Pradeep P
3 days ago
4 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 22 of 26

← Previous: Building an AI Agent → Next: Tool Calling

Layer 6 — Modern systems · Post 84 of 88

Agent memory is how an agent remembers prior steps, user preferences, and retrieved facts without stuffing the entire history into every prompt.

What you'll learn

Working memory (the context window) vs episodic vs long-term memory
Summarization, retrieval, and scratchpads — and what each is for
Why "save every message in the prompt" dies on tokens, privacy, and stale facts

The idea in one minute

The model is stateless. If you want continuity, you store state and feed a slice of it each step.

Agent memory is that store, with types:

[ Working memory ] context window: this turn's messages + tool results [ Episodic ] this session: plan, what we tried, compact summary [ Long-term ] user prefs, past tickets, retrieved facts (vector + DB)

The art is what to load. Not "everything we ever knew." That is how you hit the context limit on turn 12 and still forget the user's timezone.

Why it matters

Agents loop. Each tool observation can be 8k tokens of JSON. Ten steps later you cannot fit history, and the model drops the original goal.

Users also expect "remember I prefer tables" next week. That is not the context window. That is a database with a retrieval policy.

Interviewers will poke at context limits. "We'll use a 1M window" is not a memory design. It is a more expensive way to be unselective.

How it works

Working memory (hot)

The prompt for this model call: system instructions, the goal, recent messages, last few tool results, and any retrieved long-term snippets.

Hard cap it. Prefer structured tool results (ids, status) over pasting full HTML. If a tool returns a 50-page PDF, store it in object storage and put a pointer + excerpt in context.

Scratchpad / episodic (this run)

A short running plan and diary: "tried lookup, id=A-19, refund blocked, next=explain shipping."

You can:

Keep it as a dedicated message the model updates, or
Summarize older steps when the buffer exceeds N tokens.

Summaries lose details. Keep canonical facts (order ids, amounts) as structured fields the summary is not allowed to round-trip incorrectly.

Long-term (cold)

After the session (or during), write:

Preferences: prefer_language=en, tone=brief — small, typed, user-editable.
Episodes: embed "user asked about invoice A-19" for later semantic recall.
Entity store: order A-19 shipped 2026-08-01 in SQL, not only in prose.

On a new session, retrieve a handful of relevant memories (vector + filters), do not preload the year.

Write policy

Not every utterance is a memory. Filter: user-confirmed facts, tool-verified records. A hallucinated "user lives in Canada" must not persist. Prefer upsert with source (user said vs tool). TTL and delete-on-request for privacy (GDPR is not optional flavor text).

Isolation

Memory is per user / per tenant. Mixing memories is a leak. Same story as semantic cache, worse because you will retrieve it later as if it were true.

A simple example

Week 1: user says they use EUR and hate emojis. You store currency=EUR, style.emoji=false in a profile table — not as a 2,000-word chat log.

Week 2: they ask "what's the status of my last refund?" You retrieve profile + last refund_id from SQL. Working memory gets those two facts plus the new question. You do not replay 40 messages of small talk.

During the agent run, step 3's tool returns a 200-line audit log. You keep status=approved, eta=3d in the scratchpad and drop the log from the next prompt, with a link if the user wants the raw dump.

Common mistakes

Concatenating full history forever. Works in the demo. Then quality falls because the goal is buried, and the bill rises.

Summarizing numbers sloppily. "$199.00" becomes "about 200." Keep money and ids structured.

Vector memory without ACLs. You retrieve another customer's "API key" snippet. Partition by tenant.

Remembering the model's guesses. Only persist what you verified or the user confirmed.

One blob named memory. You cannot expire, audit, or correct it. Split profile vs session vs corpus RAG.

How this shows up in real systems

ChatGPT / Claude "memory": extracted prefs + optional recap, with user controls.
MemGPT-style / RAG-over-logs: retrieve past turns; still need a cap.
Redis for session, Postgres for profile, vector DB for episodes: a boring, correct split.
LangGraph / Temporal state: durable working memory so a retry does not amnesia the loop.

The agent loop (previous) is the clock. Memory is the state. Hands come next: tool calling.

Recap

Split working, session, and long-term memory; load a slice, not the archive.
Store structured facts for ids and money; retrieve prose with the same care as RAG.
Isolate by tenant, persist only verified or user-confirmed data, and give people a delete.

Tools are how the agent reads and changes the world outside that memory.

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 22 of 26

← Previous: Building an AI Agent → Next: Tool Calling