Designing a Metrics Pipeline

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 8 of 26

← Previous: Designing a Payment System → Next: Designing a Log Aggregation System

Layer 6 — Modern systems · Post 70 of 88

A metrics pipeline ingests huge volumes of time-series data, aggregates it, and serves dashboards and alerts without falling behind.

What you'll learn

The path from scrape or push, through aggregation, into a time-series store and alert engine
Why cardinality (unique label combinations) is the usual outage, not "not enough CPU"
How rollups and retention keep dashboards fast without storing every raw sample forever

The idea in one minute

A metric is a named number over time, with labels: http_requests_total{path="/pay",code="500"}. You collect millions of these, aggregate (sum, rate, histogram buckets), store them, and query them for graphs and alerts.

App / sidecar / node exporter | | scrape (Prometheus) or push (Datadog / StatsD) v [ Ingest / scrape pool ] --> [ Write buffer / Kafka optional ] | | v v [ TSDB: Prometheus / Mimir / Influx / CloudWatch ] | +--> [ Query API: dashboards ] +--> [ Ruler / alertmanager ] | ingest lag? drop high-cardinality series, keep RED metrics

You do not store every HTTP request as a metric. You store counters and histograms. Logs are the next post.

Why it matters

If this pipeline lags, you page late. If it explodes in cardinality, you page because the metrics system itself is down. Interviewers want pull vs push, aggregation, and cardinality in the first five minutes.

SLOs and error budgets are just queries on this data. Payments from the previous post are one of the series you will protect (charge_success_rate).

How it works

Instrumentation. Libraries expose counters, gauges, histograms. Keep label sets bounded (status code, method, not user_id).
Collection. Pull: Prometheus scrapes /metrics on a interval. Push: apps send UDP/HTTP to a gateway (StatsD, Datadog agent). Pull is simpler for long-lived servers; push fits short jobs and some multi-tenant SaaS.
Ingest. At scale, a pool of distributors hashes series to ingesters. A Kafka buffer absorbs bursts. Writes are appends of samples (series_id, timestamp, value).
Storage. A TSDB stores chunks by time. Recent data in memory/fast disk; older data compacted. Rollups: 15s raw for 24h, 5m for 2 weeks, 1h for a year — query the right resolution.
Query and alert. PromQL (or equivalent) computes rate() and histogram_quantile(). Rulers evaluate every minute. Alertmanager dedupes and routes. Dashboards hit replicas, not the ingest hot path if you can split them.
Failure. Scrape target down: you get a stale/missing series — alerts should fire on up == 0. Ingest overload: shed by series, keep golden signals (latency, traffic, errors, saturation). Never block the product API on a metrics emit (UDP or async).

Clients are Grafana and on-call. APIs are remote_write and query. Stores are the TSDB. Failure is lag and cardinality, not only disk.

A simple example

Each payments pod exposes payment_attempts_total{result="success|fail"} and a latency histogram. Prometheus scrapes every 15s. You alert: rate(fail[5m]) / rate(attempts[5m]) > 0.02. A developer adds user_id as a label "for debugging." Cardinality jumps from 20 series to 20 million. Ingesters OOM. Dashboards blank. You drop the label, keep user_id in traces or logs, and the pipeline recovers.

A cron job that lives 2 seconds cannot wait to be scraped: it pushes a completion metric to a gateway on exit.

Common mistakes

Unbounded labels (user_id, email, raw URL with IDs). This is the classic outage.

Using metrics as an event log. High-volume unique events belong in logs or traces.

Alerting on raw counters instead of rates, or on noisy 1-minute blips without for: 5m.

One giant global scrape interval for everything. Fast for SLIs, slower for batch job gauges.

Blocking request handling on a cloud metrics HTTP call. Emit async or you couple p99 to the vendor.

How this shows up in real systems

Prometheus + Alertmanager + Grafana: the default on-cluster story; Mimir/Cortex/Thanos for long-term and multi-tenant.
Datadog, CloudWatch, Stackdriver, New Relic: push agents, same cardinality physics.
OpenTelemetry metrics: the export protocol; you still need a backend.

Metrics tell you that it broke. Logs tell you which request. That is next.

Recap

Collect counters and histograms with bounded labels, aggregate, store in a TSDB.
Design rollups, alerts, and scrape/push explicitly; cardinality is the capacity plan.
Never make the product wait on ingest.

Logs are fatter, more searchable, and easier to bankrupt yourself with.

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 8 of 26

← Previous: Designing a Payment System → Next: Designing a Log Aggregation System