LLM cost management is treating tokens like a scarce resource: routing, caching, truncation, smaller models, and budgets that actually stick.

AI observability tracks quality, latency, tool errors, and cost per trace, because a 200 OK can still be a wrong or expensive answer.

Multi-agent systems split work across specialized agents that coordinate. The gain is modularity. The risk is chaos, cost, and loops.

Tool calling is how a model reaches outside itself — search, databases, APIs — in a structured way the application can actually execute.

Agent memory is how an agent remembers prior steps, user preferences, and retrieved facts without stuffing the entire history into every prompt.

An AI agent is a loop: the model plans, calls tools, observes results, and continues until the task is done or it should stop.

RAG retrieves relevant documents at query time and feeds them to the model so answers can be grounded in your data instead of only in training weights.

A vector database stores embeddings and finds nearest neighbors, which is how systems retrieve 'similar meaning' instead of exact keywords.

Semantic caching reuses a previous LLM answer when a new question is close enough in meaning, which can cut both latency and spend.

AI rate limiting is not just requests per second. Tokens, GPU-seconds, and cost per user all have to be capped or someone will melt the bill.

Model routing picks which model should handle a request based on cost, quality, latency, or task type — a load balancer for intelligence.

GPU scheduling decides which model, which batch, and which request gets the scarce accelerators next. Waste here is measured in dollars per minute.

LLM request queues absorb bursts, enforce fairness, and keep GPUs busy without overloading them into timeouts.

Token streaming is the protocol-level idea: the model emits pieces of text over time, and every layer from GPU to browser has to stay in sync with that flow.

Streaming sends tokens to the client as they are produced so users see an answer forming instead of staring at a spinner.

Disaster recovery is the plan for losing a datacenter, a region, or a whole cloud account — and still bringing the business back within an agreed time.

An LLM inference server loads a model and runs generation requests. It is the new 'app server', except the unit of work is tokens on a GPU.

Active-active serves traffic from multiple sites at once. Active-passive keeps a standby ready. The difference is latency, cost, and conflict handling.

Multi-region architecture runs the system in more than one geographic place so users are closer and a whole region can disappear without taking you down.

A file storage system stores blobs reliably, serves them quickly, and handles uploads, metadata, permissions, and virus-scan-sized side jobs.