Building an AI Agent

Pradeep P
3 days ago
4 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 21 of 26

← Previous: RAG Architecture → Next: Agent Memory

Layer 6 — Modern systems · Post 83 of 88

An AI agent is a loop: the model plans, calls tools, observes results, and continues until the task is done or it should stop.

What you'll learn

The agent loop vs a single LLM call, in a picture you can draw on a whiteboard
Stop conditions, step budgets, and why "just let it think" is not a design
When a RAG pipeline is enough and when you actually need an agent

The idea in one minute

A chat completion is one shot: prompt in, tokens out. An agent is a state machine the model drives:

goal + memory | v [ model: think / pick action ] | +-- respond to user → stop +-- call tool → observe result → loop +-- ask human → pause

The application executes tools, appends observations, and calls the model again. The model never "runs Python" by itself unless you built that sandbox on purpose.

If you cannot draw the loop and the exit, you do not have an agent. You have a prompt with aspirations.

Why it matters

Users ask for work, not paragraphs: "file a ticket and email me the id," "compare these three PDFs," "debug this failing job." That needs act, see, act again.

The cost is real: each round is tokens + latency + tool risk. Unbounded loops are how you discover your TPM limit at 3 a.m.

In interviews, "we'll use an agent" is fashionable. The follow-ups are tools, memory, termination, and idempotency. This post is the skeleton; the next three hang meat on it.

How it works

The loop

State: user goal, conversation, tool results, maybe a plan.
Model step: produce a user-visible message or a structured tool call (post 85).
Execute in your process: timeouts, authz, retries.
Observe: tool JSON / error string goes back into state.
Repeat until a stop rule fires.

Patterns you will hear: ReAct (reason + act), plan-then-execute, tool-calling APIs that skip the prose. The architecture is the same. The prompt is not.

Stop rules (non-negotiable)

Max model steps (e.g. 8) and max wall clock.
Max tokens / dollars for this run (ties to posts 79 and 88).
Model emits a final answer.
Tool reports a terminal success/failure you trust.
User cancel / human reject.

If the model keeps calling search with the same query, break. Detect loops: hash recent tool calls; on repeat, stop or switch strategy.

Control plane vs model

You own: which tools exist, sandboxing, what "done" means, retries, and what is shown to the user. The model owns: which legal tool to try next, given the observations.

Do not let the model invent a tool name. Do not let it pick delete_everything without a policy. That is product, not intelligence.

Agent vs RAG vs workflow

RAG: one retrieve, one generate. Use it when the answer is in documents.
Fixed workflow: you already know the steps (validate → charge → email). Use a job runner. Do not pay an LLM to be Airflow.
Agent: the branching is unknown: search or SQL or both, then maybe a ticket.

Start simpler. Agents are for uncertainty in the plan, not for résumé keywords.

A simple example

"Find the order for email x, refund it if it is still pending, otherwise explain why not."

Step 1: model calls lookup_order(email). Observe: status shipped, id A-19. Step 2: model should not call refund. It explains shipping policy. If status had been pending, step 2 is refund(A-19) with an idempotency key, then a user message.

Caps: 5 steps, 30 s, refund tool requires the order to be in pending in your code, not only in the model's head. If lookup fails twice, stop and say so.

A one-shot prompt cannot refund. A naive loop without the status check in code will refund shipped orders because the model was optimistic.

Common mistakes

No step budget. The demo works on one happy path. Production meets a confused model and a recursive search.

Tools that are too wide. run_sql(any) is an incident. Narrow tools: get_order(id), parameterized.

Hiding errors. "Tool failed" with no body trains the model to hallucinate a result. Return structured errors.

Agent for a form. If the user is filling known fields, use a form.

Zero human checkpoint on irreversible actions. Draft the refund, confirm, then execute.

How this shows up in real systems

OpenAI Assistants / Responses, Claude tool use, Gemini function calling: the loop is still yours if you run tools server-side.
LangGraph, Temporal + LLM steps: durable state so a crash does not forget the plan.
Support and coding agents: the successful ones look boring: few tools, tight schemas, hard stops.

Memory (next) is how the loop does not shove the entire internet into every prompt. Tools (the post after) are the hands.

Recap

An agent is a bounded loop: model → tool or answer → observe → repeat.
You enforce budgets, authz, and stop conditions; the model only chooses among allowed actions.
Prefer RAG or a workflow until the plan is genuinely unknown.

The loop needs a place to put what it has already seen. That is agent memory.

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 21 of 26

← Previous: RAG Architecture → Next: Agent Memory