LLM Cost Management

Pradeep P
3 days ago
4 min read

Updated: 2 days ago

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 26 of 26

← Previous: AI Observability → Next: Fallacies of Distributed Computing

Layer 6 — Modern systems · Post 88 of 88

LLM cost management is treating tokens like a scarce resource: routing, caching, truncation, smaller models, and budgets that actually stick.

What you'll learn

How the bill is actually computed (tokens × price × retries × agent steps)
Levers that work: route, cache, shrink context, cap output, batch, and stop
How to attach hard budgets to tenants so dashboards are not a wish

The idea in one minute

LLM spend is not "the API subscription." It is Σ (tokens × unit price) plus GPU idle time if you self-host — multiplied by hidden loops (retries, tool rounds, cascades).

Cost management is making that sum a first-class SLO, the way you already treat latency:

Request
  → semantic / prefix cache?     (tokens = 0 or tiny)
  → route to smallest OK model
  → trim retrieval and history
  → cap max_tokens, stop sequences
  → generate
  → record $ on the trace
  → enforce tenant budget

If you only optimize the model and never the path, you will pay frontier prices for "hi."

Why it matters

Unit prices drop; usage does not. Agents and RAG multiply calls. A feature that is "just one completion" in the design doc is twelve in production.

Finance will not accept "GPUs are the strategy." Interviewers will not either. They want routing, caching, quotas, and a number (cost per successful task), plus what you sacrifice (quality, latency) to hit it. Every prior AI post in this stretch is a lever: queues, GPUs, routing, limits, cache, RAG, agent budgets, traces.

How it works

Know the formula

Hosted: input_tokens * pin + output_tokens * pout (output often dearer). Add image tokens, cached-prefix discounts, and your retries. Self-hosted: $/GPU-hour / useful tokens. Idle GPUs are a cost bug (post 77).

Attribute cost to tenant, feature, model, prompt_version. Averages hide the agent that loops.

Levers, in the order you should pull them

Do not call the model. Semantic cache, exact cache, deterministic code for "what's 2+2" class tasks.
Call a cheaper model. Routing and cascades (post 78). Most turns are easy.
Send fewer tokens in. Tight system prompts, summarized memory (post 84), reranked RAG with a small k, no raw HTML from tools.
Provider prompt/prefix caching for a stable system prompt. This is not semantic cache; it is a discount on repeated prefixes.
Cap tokens out. max_tokens, stop sequences, "answer in 3 bullets." Users can ask for more.
Batch and off-peak APIs for evals and offline jobs. Do not use interactive GPUs for nightly embeds.
Stop the loop. Agent step caps, no retry-on-429 without a budget (post 79).

Budgets that stick

A dashboard is not a budget. Implement:

Soft alert at 70% of daily tenant cap.
Hard 429 / feature disable at 100%.
Separate caps for eval keys vs production.
A kill switch for a prompt version that regresses cost/task.

Reserve cost the way you reserve TPM: worst-case max_tokens on admit for streaming.

Quality is the constraint

Cutting k retrieved chunks saves money and can hurt faithfulness. Measure cost per successful graded task, not cost per 1k tokens in isolation. Otherwise you will "save" your way into hallucinations.

A simple example

Your copilot is $0.42 per thread. Target is $0.12.

Traces show: 8k-token policy dump in every prompt, GPT-class model for "reset password," agents averaging 9 steps, 12% exact-repeat questions uncached.

You: (1) exact + semantic cache on public FAQs, (2) route reset_password to 8B, (3) rerank to 3 chunks, (4) cap agent steps at 5, (5) prefix-cache the 2k-token system prompt. Cost/thread falls to $0.11. Faithfulness eval stays within 2%. The remaining spend is the long-tail of hard billing questions — those should hit the expensive model.

A team that only "switched to a cheaper model for everything" would have missed the cache and wrecked billing answers.

Common mistakes

Optimizing list price, ignoring steps. A cheap model called 40 times loses to a mid model called twice.

Unbounded max_tokens. One "write a book" user is a line item.

Retries without a cap. Your outage becomes a second bill.

No tenant attribution. You cannot price the product.

Context stuffing instead of retrieval. You pay to ignore most of the prompt. Idle owned GPUs are waste too — pack work (post 77) or shut nodes down.

How this shows up in real systems

Provider invoices + your traces (Helicone, OpenMeter, custom BigQuery): you need both.
OpenAI batch / Bedrock provisioned throughput: different unit economics for offline vs chat.
FinOps for ML: same as cloud FinOps, with tokens as the SKU.
Product packaging: free tier is a cost control (post 79), not only marketing.

Observability (previous) shows where the money went. This post is the policy that uses those traces.

Recap

Cost is tokens × price × hidden loops; attribute it per tenant and feature.
Avoid calls, then route down, then shrink context and caps, then pack GPUs; put hard budgets on the path.
Optimize $ per successful task so cheap does not mean wrong.