AI Observability

Pradeep P
3 days ago
4 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 25 of 26

← Previous: Multi-Agent Systems → Next: LLM Cost Management

Layer 6 — Modern systems · Post 87 of 88

AI observability tracks quality, latency, tool errors, and cost per trace, because a 200 OK can still be a wrong or expensive answer.

What you'll learn

Why HTTP 200 is not success for an LLM, and what a trace must contain
Latency (TTFT vs tokens/s), tokens, cost, tool errors, and quality signals
How evals and production logging work together without leaking PII

The idea in one minute

Classic observability: logs, metrics, traces. AI systems add a nasty twist: the response can be fluent and wrong, or right and ruinously expensive.

AI observability means every user-facing answer is a trace of the pipeline:

trace_id ├─ retrieve (ids, scores, latency) ├─ model call (model_id, prompt version, tokens in/out, TTFT, $) ├─ tool calls (name, args hash, duration, error) └─ output (answer, citations, user feedback)

If you cannot open one id and see why it said that, you are flying on vibes.

Why it matters

You will ship a prompt change that looks better on three examples and tanks faithfulness on RAG. You will have a week where latency is fine but cost per chat doubled because routing shifted to the frontier model. You will have a tool timeout that the model covered with a confident story.

Interviewers expect OpenTelemetry-style thinking plus token and quality dimensions. "We'll check CloudWatch 5xx" is how you miss the incident.

How it works

Metrics (the dashboard)

Traffic: requests, cache hits (semantic + prefix), 429s.
Latency: TTFT (time to first token), time-to-complete, queue wait vs generation (post 76).
Tokens: prompt, completion, cached vs billed.
Cost: estimated $ per route / tenant / model.
Tools: error rate, p95 duration.
Quality proxies: thumbs, regeneration rate, "I don't know" rate, citation coverage.

Alert on cost and TTFT, not only on errors. A silent switch to a bigger model is a SEV without a 500.

Traces (the debugger)

One trace_id from gateway to GPU. Spans for retrieve, rerank, each model hop, each tool. Attributes: model_id, prompt_version, temperature, retrieved doc_ids, agent step index, agent name.

Store the prompt carefully: it is PII, secrets, and copyrighted docs. Redact; sample; encrypt; retain with a TTL. You still need enough to reproduce (prompt template + variables, not always the raw dump).

Quality is a pipeline

Offline evals: a frozen set of questions with graded answers / required citations. Run on every prompt and routing change. Split retrieval recall vs answer faithfulness vs style.

Online: sample traces for human review; LLM-as-judge is a metric, not ground truth. Use it to find regressions, then look at the trace.

User thumbs are biased (they downvote latency). Join them with traces anyway.

Prompt and index versions

Tag every call with prompt_version and index_build. Otherwise "quality dropped Tuesday" has no bisect. This is config management for English.

A simple example

p95 complete time jumped from 2 s to 9 s. Error rate is flat.

You slice traces: queue wait is 6 s (not the model). GPU scheduler is packed with a new batch job on the interactive pool. You move batch. TTFT recovers.

A week later thumbs-down spikes. Traces show retrieve returning a deprecated policy doc_id=.../v3 after a failed ingest. Faithfulness eval on CI would have caught it if the eval set included that question. You add the case, fix ingest, and alert when citation doc version is older than a threshold.

Without traces you would have "tuned the prompt" for a week.

Common mistakes

Logging only the final string. You cannot tell retrieve vs model vs tool.

No model_id / prompt_version. You cannot A/B or explain cost.

Treating 200 + JSON parse success as quality. Valid JSON can still be a fabricated order id.

Storing raw prompts forever in a shared Slack exporter. Compliance incident.

Evals only on happy-path poetry. Include "should refuse," injection, and missing-doc cases.

Metrics without tenant labels. One customer burns 80% of tokens; your average looks fine.

How this shows up in real systems

LangSmith, Langfuse, Helicone, Phoenix, Braintrust: productized LLM tracing and evals.
OpenTelemetry + GenAI semantic conventions: the portable version.
Provider dashboards: necessary, not sufficient — they do not see your RAG or tools.
Classic APM (Datadog, Grafana): still the home for infra; add token and quality panels.

Multi-agent graphs (previous) are unreadable without per-node spans. The last post turns those $ attributes into a discipline: cost management.

Recap

A successful LLM request is correct, fast enough, and within budget — not merely HTTP 200.
Trace retrieve, model, tools with versions, tokens, and ids; redact PII.
Pair offline evals with production traces; alert on cost and TTFT as well as errors.

You cannot manage what you do not meter. Tokens are the unit of money. That is the end of the series.

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 25 of 26

← Previous: Multi-Agent Systems → Next: LLM Cost Management