Pub/Sub

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 2 — Communication

Layer 2 · Post 6 of 10

← Previous: Kafka → Next: Event-Driven Architecture

Layer 2 — Communication · Post 21 of 88

Pub/Sub lets a publisher announce an event once and many subscribers react independently, without knowing about each other.

What you'll learn

How publish/subscribe differs from a competing-consumer queue
Topics, subscriptions, and fan-out
What you give up: coupling to a broker, extra latency, harder tracing

The idea in one minute

Publish/subscribe: a producer publishes a message to a topic. Every subscriber that cares gets its own copy.

A queue: many workers, one of them does the job. Pub/sub: many subscribers, each does their job.

      [ topic: user.signup ]
       /        |        \
 email       fraud      analytics

The signup service does not import the email SDK. It publishes UserSignedUp. New subscribers can appear without a deploy of the publisher.

SNS, Google Pub/Sub, Redis Pub/Sub, NATS, Kafka consumer groups (log-shaped pub/sub) are all this idea with different durability.

Why it matters

Fan-out is how products grow features without a god-service. Notifications, billing, CRM, and ML features all want the same business event.

It is also how you accidentally create a distributed monolith: twenty subscribers, no owner of the event schema, and a change to UserSignedUp pages everyone.

How it works

Topic vs subscription

Topic: the named feed.
Subscription: a subscriber's durable (or not) handle. In Google Pub/Sub, each subscription gets all messages. In Redis Pub/Sub, if you were offline you missed them.

Durability is the fork:

Transient (Redis Pub/Sub): live listeners only.
Durable (SNS+SQS, Google Pub/Sub, Kafka groups): catch up after downtime.

For anything you cannot lose, you want durable subscriptions.

Push vs pull

The broker pushes HTTP to you, or you pull. Push is easy until you fall behind. Pull (or a queue per subscriber) lets you control rate — backpressure, Layer 3.

Filtering

Some systems filter on attributes (eventType=paid) so subscribers are not flooded. Filtering in the app is simpler and wasteful. Filtering in the broker is nicer at scale.

A simple example

InvoicePaid publishes once.

Tax service records it.
Receipt service emails PDF.
Entitlements service unlocks the plan.

If receipts are down, tax and entitlements still run (durable subs). If this were one sync call from billing to all three, receipts being down could block unlocking.

Common mistakes

Redis Pub/Sub for money. It is a firehose, not a queue. Use a durable system.

Circular publishes. A updates B's topic, B publishes back, infinite loop. Name events as facts that happened, not commands, to reduce cycles (next post).

No schema, no version. data as a free JSON blob will bite you.

One mega-topic. everything with filters in every client. Split by bounded context.

How this shows up in real systems

SNS → SQS: AWS fan-out (pub/sub + a queue per worker).
Google Pub/Sub: native subscriptions.
Kafka: pub/sub via consumer groups on a log.

If you need replay six days later, you are in Kafka territory. If you need "email this once and forget," a queue might be enough.

Recap

Pub/sub fans out one publication to many independent subscribers.
Prefer durable subscriptions for work you cannot lose.
Publishers should not know subscriber lists — but someone must own the event contract.

That contract-plus-facts style, taken further, is event-driven architecture.

Series: Modern System Design · Layer 2 — Communication

Layer 2 · Post 6 of 10

← Previous: Kafka → Next: Event-Driven Architecture