Observability
- Pradeep P
- 3 days ago
- 3 min read
Layer 5 · Post 6 of 12
← Previous: Distributed Tracing → Next: Configuration Management
Layer 5 — Modern infrastructure · Post 56 of 88
Observability is the ability to ask new questions of a running system using logs, metrics, and traces — not just the dashboards you planned for.
What you'll learn
How logs, metrics, and traces answer different questions — and why you want all three
The difference between monitoring (known failures) and observability (unknown ones)
Cardinality, SLOs, and why "more dashboards" is not a strategy
The idea in one minute
Monitoring is a checklist you wrote in advance: CPU > 90%, error rate > 1%, disk full. Observability is whether the system still explains a failure you did not put on that list.
You get there with three signals:
Metrics: numbers over time (QPS, p99, saturation). Cheap, aggregatable, great for alerts.
Logs: discrete events with context. Great for "what exactly happened to order 8821."
Traces: one request across services (previous post). Great for "where did the 2s go?"
Alert (metric) --> "payments p99 is 2s" Trace --> "the /charge span is 1.8s" Log --> "deadlock on orders row 8821"
If you only have one pillar, you guess with the other two.
Why it matters
Distributed systems fail in combinations. A cache stampede plus a slow dependency plus a bad deploy is not a CPU graph. Interview designs that "add monitoring" without saying what you would look at read as incomplete.
Observability is also a product constraint: you cannot operate a mesh, autoscaler, or multi-region setup you cannot see.
How it works
Metrics are usually time series: a name, labels, a value. Prometheus scrapes /metrics; many clouds push to a vendor. You alert on SLIs (what the user felt: success rate, latency) more than on CPU. CPU is a clue, not an SLO.
Logs should be structured (JSON), have a request or trace ID, and ship to a store you can query (ELK, Loki, Cloud Logging). Unstructured stdout on a node is not a plan.
Traces we covered. The join key is the trace ID (and often request_id) on logs and spans so you can jump.
Exemplars glue metrics to traces: this p99 bucket has a link to a slow trace. That is the modern happy path.
SLOs, not vibes
An SLO is a target on an SLI: 99.9% of checkouts succeed in 300ms over 30 days. An error budget is the remainder. Burn the budget fast, page humans. That is how you avoid paging on every blip and sleeping through a real outage.
A simple example
Checkout error rate jumps from 0.1% to 3%. The metric pages you. You open the RED dashboard (Rate, Errors, Duration). Errors are 500s from payments. You click an exemplar trace, land on a span, then logs: a config flag flipped and disabled a payment provider, with no fallback. You revert the flag. The metric recovers. That loop — metric → trace → log — is observability in practice.
Without traces you SSH and grep. Without metrics you notice when Twitter notices. Without logs you know where but not why.
Common mistakes
Alerting on CPU. Users do not feel CPU. They feel errors and latency. Alert on SLIs; use CPU to debug.
Unbounded cardinality. A metric label for user_id or a log field for full request bodies will melt Prometheus or the log bill. Keep labels low-cardinality; sample bodies.
Dashboards as a substitute for on-call design. Forty graphs and no SLO means nobody knows what "broken" is.
Only collecting from the happy path. Health checks that hit a cached ok while the real dependency is dead. Probe the real path, or you will observe fiction.
How this shows up in real systems
Prometheus + Grafana, Datadog, CloudWatch, Google Cloud Monitoring: metrics and dashboards.
Loki, Elasticsearch, Splunk: logs.
OpenTelemetry: one pipeline into all three backends.
On-call: runbooks that start from the SLO dashboard, not from "check the boxes."
Layer 6's "design a metrics pipeline" and "log aggregation" posts are the build-out of these ideas as systems of their own.
Recap
Observability is asking new questions of logs, metrics, and traces — not a wallpaper of graphs.
Metrics alert, traces locate, logs explain. Join them with IDs.
Drive pages from SLIs/SLOs, and watch cardinality like a budget.
Once you can see the system, you still have to change its behavior safely. That is configuration.
Layer 5 · Post 6 of 12
← Previous: Distributed Tracing → Next: Configuration Management
Comments