Designing a Log Aggregation System

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 9 of 26

← Previous: Designing a Metrics Pipeline → Next: Designing a File Storage System

Layer 6 — Modern systems · Post 71 of 88

A log aggregation system collects logs from everywhere, makes them searchable, and keeps cost from exploding as traffic grows.

What you'll learn

The collect → buffer → index → query pipeline, and why the buffer is not optional at scale
Why structured logs and sampling beat "log everything as text"
How retention tiers (hot search vs cold object storage) keep the bill from tracking QPS linearly

The idea in one minute

Every pod writes lines. You ship them off the node, buffer so a search cluster blip does not stall apps, parse into fields, index what you must search, and expire the rest.

App stdout / file | v [ Agent: Fluent Bit / Vector / OTel ] --> [ Kafka / Pub/Sub ] | | | ship fail? local disk buffer v | [ Indexer: ES / Loki / Cloud ] v | never block the request on the agent +--> hot search (7d) +--> S3/GCS compact (90d) | cluster full? drop debug, keep error

Metrics are numbers. Logs are events with text and fields. You query them when an alert fires: "show me payment_id=pay_1 around 10:03."

Why it matters

Without central logs, you SSH into boxes — and you cannot do that in Kubernetes. With unbounded logs, you pay more for Elasticsearch than for the product. Interviewers want backpressure, parsing, and retention, not "we'll use ELK."

PII in logs is a compliance incident. Payment payloads in debug logs are how you fail PCI without trying.

How it works

Clients are actually agents on each node (or sidecar). They tail files or stdin, attach pod, service, region. Apps should log JSON (level, msg, trace_id, payment_id) so you do not regex a snowflake format per team.
Buffer. Kafka (or equivalent) decouples shippers from indexers. If OpenSearch is slow, the queue grows; agents use local disk buffers and drop or sample debug first, never block the JVM request thread on a full socket.
Parse and enrich. Indexers grok/JSON-parse, geo-IP, redact secrets (Authorization, card numbers). Bad parse goes to a dead-letter index, not the floor.
Index vs store. Elasticsearch-style: inverted index on fields you search — expensive. Loki-style: index labels only, grep chunks in object storage — cheaper, slower ad hoc search. Many shops: errors in a hot index, the rest in cheap storage.
Query. Kibana/Grafana explore, plus deep links from traces (trace_id). Multi-tenant: enforce team= filters like search tenant filters.
Failure. Indexer down: buffer. Buffer full: prioritize level>=error. Agent down: you lose that node's logs — detect with a heartbeat metric (log_shipper_up).

Sampling: keep 100% of errors, 1% of successful request logs, 100% of sampled traces. Volume follows traffic; cost should not unless you choose that.

A simple example

Checkout logs {"level":"info","msg":"charge_ok","payment_id":"pay_1","latency_ms":212}. Fluent Bit ships to Kafka. Indexers write to a 7-day hot index. On-call gets a metrics alert, queries payment_id:pay_1, sees the line and a timeout to the PSP. Debug logs of full Stripe responses were sampled away — good, because they contained tokens. After 7 days the JSON lives in compressed S3; you can rehydrate for an audit, not for a 50ms Kibana query.

A new service logs the entire inbound body. Cardinality of unique messages explodes, shards hit max map count, search slows. You add a allowlist of fields and a size cap per line.

Common mistakes

Synchronous HTTP to the log vendor inside the request. Your p99 becomes their p99.

Unstructured printf with no trace_id. Correlation with metrics and traces dies.

Infinite retention in a hot cluster. Use ILM: hot → warm → delete/S3.

Logging PII and secrets. Redact at the agent. Treat logs as a data store with access control.

Indexing every field "just in case." Mapping explosion. Index what you query; store the rest.

How this shows up in real systems

ELK / OpenSearch, Splunk, Datadog Logs, Cloud Logging: hosted or self.
Grafana Loki, ClickHouse log tables: cheaper patterns for high volume.
OpenTelemetry logs: same pipeline, standard export.

You now have numbers and lines. Users still need to store files — blobs, not log lines.

Recap

Agent → buffer → index/store → query, with drop policy when full.
Structured logs, sampling, and tiered retention are the design, not afterthoughts.
Never block user requests on shipping, and never index unbounded fields.

File storage is the other "keep bytes forever, serve them fast" problem.

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 9 of 26

← Previous: Designing a Metrics Pipeline → Next: Designing a File Storage System