Distributed Tracing

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 5 — Modern infrastructure

Layer 5 · Post 5 of 12

← Previous: Service Meshes → Next: Observability

Layer 5 — Modern infrastructure · Post 55 of 88

Distributed tracing follows one request across many services so you can see where time was spent and where it broke.

What you'll learn

What a trace and a span are, and how a trace ID travels
Why logs per service are not enough once a request fans out
Sampling, context propagation, and the mistakes that break the chain

The idea in one minute

A checkout click touches the API gateway, checkout, payments, inventory, and a queue. Each service has logs. None of them know they are the same request.

Distributed tracing stamps a trace ID on the request and records timed spans at each hop. You open one waterfall: where the 2 seconds went, and which span threw.

[ gateway 12ms ] └── [ checkout 80ms ] ├── [ inventory 40ms ] └── [ payments 1.8s ] <-- the smoking gun

Why it matters

Microservices without tracing turn every incident into a meeting: "is it us or you?" Latency is not one number; it is a path. p99 of 2s with a 1.8s payments span is a different design conversation than 2s of GC on checkout.

Interviewers like this because it shows you operate the system, not just box-draw it. Observability (next post) is the umbrella; tracing is the request-shaped piece.

How it works

The first edge (gateway, mesh, or app middleware) creates a trace and a root span.
Context (trace ID, parent span ID, sampling flag) is sent onward: HTTP headers (traceparent in W3C Trace Context), gRPC metadata, or message attributes on a queue.
Each service starts a child span, times work, records status and attributes (http.route, db.statement hashed, user id if you must).
A collector (OpenTelemetry Collector is the usual) receives spans, and a backend (Jaeger, Tempo, Zipkin, Honeycomb, Datadog, X-Ray) stores and queries them.

Sampling is how you survive. Head sampling decides at the start (1% of traces). Tail sampling keeps interesting traces (errors, slow ones) after the fact. 100% of spans in a busy fleet is a cost and a privacy problem.

What a span is not

A span is not a log line. It has a start, a duration, and a parent. Logs still explain why; the trace tells you where to look. Metrics tell you how often. You want all three — next post.

A simple example

Users say checkout is slow. Metrics show p99 up. You filter traces for checkout with duration > 1s. Almost every slow trace has POST /charge at 1.7s, and that span's child is a Postgres query missing an index. You never grepped three log pipelines hoping the timestamps lined up.

A worker consumes order.created. If the producer did not copy traceparent onto the message, the worker starts a new trace. You lose the story across the async hop. Propagation is the whole game.

Common mistakes

No propagation on queues. HTTP is instrumented; Kafka is not. The most painful bugs live in that gap.

Logging the trace ID but never creating spans. You can grep, but you cannot see a waterfall or compare siblings.

Attributes that explode cardinality or leak PII. A span per user-email value wrecks the backend and your compliance story. Use low-cardinality keys; hash or drop secrets.

100% sampling in production "just for a week." It becomes the bill. Start with errors + a small random fraction.

Instrumenting only your code, ignoring the mesh/gateway. Then the missing 200ms is "the network." Instrument edges too.

How this shows up in real systems

OpenTelemetry: the vendor-neutral SDK and collector. This is the default answer now.
Jaeger / Grafana Tempo / Zipkin: self-hosted backends.
Datadog, Honeycomb, New Relic, Cloud X-Ray: SaaS. Same model: traces in, waterfalls out.
Service meshes: auto-span at the proxy. Great for hop timing; you still need app spans for DB and business logic.