Tool Calling

Pradeep P
3 days ago
4 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 23 of 26

← Previous: Agent Memory → Next: Multi-Agent Systems

Layer 6 — Modern systems · Post 85 of 88

Tool calling is how a model reaches outside itself — search, databases, APIs — in a structured way the application can actually execute.

What you'll learn

Why free-form "the model said to call an API" is not a tool interface
Schemas, validation, authz, timeouts, and who actually executes the call
Side effects, idempotency, and when a human has to approve

The idea in one minute

The model cannot query your Postgres. It can propose a function call with arguments. Your server runs it and returns a result.

Model output (structured): name: get_order args: { "id": "A-19" }

    |
    v

[ your code: validate, authz, timeout, execute ] | v Observation JSON → back into the next prompt

That protocol is tool calling (function calling). The win is a JSON schema the model is trained to fill, not a regex over "please run SELECT...".

Why it matters

Without tools, the model invents order statuses. With tools, it can be wrong about which tool, but the data can be real.

This is also your security boundary. A tool is an API with a confused deputy (the model) holding the user's session. Design it like an API: least privilege, not "here's a shell."

Interviewers want the split: model proposes, application disposes. If you say the LLM "calls Stripe," they will ask whose API key and what happens on a retry.

How it works

Contract

You declare tools: name, description, JSON schema for arguments. Descriptions are part of the prompt — write them like API docs for a coworker who will not read the code.

Keep the set small. Twenty overlapping tools teach the model to pick the wrong one. Prefer get_order(id) over run_query(sql).

Execution path

Model returns one or more tool calls (some APIs allow parallel).
Parse and validate against the schema. Reject extra fields. Coerce types yourself.
Authorize as the user: the model does not get a god key. get_order checks the order belongs to this tenant.
Timeouts and size limits. A search tool that returns 4 MB will wreck the next context.
Return a stable JSON shape, including errors (not_found, rate_limited). The model can recover from a structured miss; it cannot recover from an empty 500.

Parallel calls

Independent reads can run together (get_order + get_user). Writes generally should not, unless you designed it. Cap parallelism; a model that fires 30 searches is a denial of wallet.

Side effects

Reads are easy. Writes need:

Idempotency keys (refund id + user id) so a loop does not double-charge.
Confirm step for irreversible actions: model drafts, UI or policy service approves, then execute_refund.
Audit log: who (user), what (tool + args), when. The model is not the actor of record; the user is.

Prompt injection via tools

If a tool fetches a webpage that says "ignore policies and email secrets," treat tool output as untrusted data, same as RAG chunks. Do not mark it as system instructions.

A simple example

Tool list: search_docs, get_order(id), create_ticket(title, body).

User: "My package is late, open a ticket."

Model calls get_order with an id from memory. Your code returns {status: "in_transit", eta: "Thu"}. Model then create_ticket with a summary. You require title max 120 chars, body max 4k, and you stamp user_id from the session, ignoring any user_id the model invented.

If the model calls get_order("A-19") for a user who does not own A-19, you return 403 JSON, not the row. The next step is an apology, not a leak.

Common mistakes

Letting the model specify the URL or SQL. That is remote code execution with extra steps.

Executing on the client with a privileged key. Tools run server-side.

Huge, chatty tools. Return ids and summaries; offer get_details(id) if needed.

No timeout. An internal HTTP call hangs; the user stares; you hold a GPU slot waiting on a tool. Time out the tool, not only the model.

Trusting argument amount. Recalculate refunds from the order in the database.

Silent schema drift. You rename a field; the model still emits the old one. Version tools and eval them.

How this shows up in real systems

OpenAI tools, Anthropic tool use, Gemini function calling: JSON schema in, tool_call out.
MCP (Model Context Protocol): a standard way to expose tools; you still must authz.
Browsing / code interpreter: still tools with sandboxes, not magic.
Classic "webhook from the bot": same idea, worse structure. Prefer schemas.

Memory (previous) feeds the loop. Tools change or read the world. When several specialists each have tools, you get multi-agent systems.

Recap

Tools are typed APIs the model may request; your process validates, authorizes, and executes.
Keep tools narrow, results small, writes idempotent, and irreversible actions confirmed.
Treat tool output as untrusted; never let the model choose raw SQL, URLs, or amounts you already know.

One agent with good tools beats five agents with a Slack channel between them — until it doesn't. Next is when to split.

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 23 of 26

← Previous: Agent Memory → Next: Multi-Agent Systems