Kafka

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 2 — Communication

Layer 2 · Post 5 of 10

← Previous: Message Queues → Next: Pub/Sub

Layer 2 — Communication · Post 20 of 88

Kafka is a durable, ordered, replayable log. Consumers read at their own pace, which makes it a backbone for event-driven systems.

What you'll learn

How a log differs from a queue
Topics, partitions, offsets, consumer groups
Why replay and fan-out make Kafka a backbone, not just a buffer

The idea in one minute

A queue forgets a message after a consumer acks it. A log keeps the message for a retention period. Many consumers can read the same data, each with their own offset ("I have read up to here").

Apache Kafka is the widely used implementation: producers append to topics, topics are split into partitions for scale, consumers in a group split partitions among themselves.

You can rebuild a projection by rewinding. You can add a new service that reads last week's events. That is a different superpower than SQS.

Why it matters

Modern companies dump facts into a log: OrderPlaced, PaymentCaptured, InventoryReserved. Search, analytics, fraud, and email all subscribe. The producer does not know who will exist next year.

That is also why Kafka ops is a job: disks, consumer lag, partition count, and "we compacted the wrong topic" are real incidents.

How it works

Anatomy

Topic: a named stream (orders).
Partition: an ordered slice of the topic. Order is per partition, not global.
Offset: the position in a partition.
Producer: appends records, often with a key so the same user always hits the same partition (ordering per user).
Consumer group: a team of processes sharing work. Each partition is assigned to one member of the group. A second group reading the same topic gets its own copy of the stream (fan-out).

Topic "orders" partition 0: r1 r2 r5 <- consumer A partition 1: r3 r4 r6 <- consumer B

Another group (analytics) also reads 0 and 1 from the start or from now.

Retention and compaction

Time/size retention: delete old segments. Compaction: keep the latest value per key (useful for changelog streams, not for "every click").

Compared to a queue

Queue after consume: usually gone. Kafka: still there.
Many independent readers: extra queues / pubsub vs extra consumer groups.
Replay: no vs yes.
Order: optional / FIFO-key vs per partition.

Use a queue for tasks ("do this job once"). Use Kafka for facts ("this happened; whoever cares can listen, even later").

A simple example

Checkout writes OrderPlaced to Kafka (key = orderId).

Email service group: sends mail, commits offsets.
Warehouse group: prints pick lists.
Fraud group: scores the order. If they deploy a new model, they reset offsets and reprocess yesterday.

The checkout API does not call those three. It writes one event.

If email is down, lag grows. Events wait on disk. That is the point.

Common mistakes

One partition "for simplicity." Throughput and parallelism die; one consumer, one disk.

Bad keys. All events keyed null or keyed restaurantId when one restaurant is 80% of traffic → a hot partition.

Treating Kafka as a task queue without thinking about retries and poison. You can, with care (retry topics). A DLQ pattern is still needed.

Unbounded payload / no schema. Schema registry (Avro/Protobuf/JSON Schema) saves you from silent poison.

Ignoring consumer lag. Lag is the product metric. If warehouse lag is 3 hours, the log is working and the business is not.

How this shows up in real systems

LinkedIn (origin), Netflix, Uber, almost every large event platform.
MSK / Confluent Cloud: managed Kafka.
Redpanda, Pulsar: similar log idea.

Layer 4's CDC often lands in Kafka. Layer 6's metrics/logs pipelines look like this too.

Recap

Kafka is a durable, partitioned, replayable log.
Consumer groups fan out independently; keys pick partitions and ordering.
Use it for events you may need again, not as a dump for one worker.