Designing a Notification System

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 1 of 26

← Previous: Disaster Recovery → Next: Designing a Real-Time Chat System

Layer 6 — Modern systems · Post 63 of 88

A notification system accepts events from many products and delivers email, push, SMS, or in-app messages at scale without spamming or dropping the important ones.

What you'll learn

How an ingest API, a queue, and per-channel workers separate "something happened" from "deliver this"
Why preferences, rate limits, and idempotency keys matter more than picking Twilio vs SES
What fails when a worker crashes mid-send, and how you recover without double-texting

The idea in one minute

Products emit events ("order shipped", "mention in a comment"). A notification service turns each event into one or more deliveries on channels the user actually wants: email, push, SMS, in-app.

You do not call APNs from the checkout service. You publish an event, then a pipeline decides who, which channel, which template, and when.

The hard part is not sending one email. It is fan-out, preference, and "do not spam" at millions of events per day.

Why it matters

Interviewers use this prompt because it mixes queues, fan-out, third-party APIs, and user trust. A "password reset" must land. A "we miss you" email must not wake someone at 3 a.m. five times.

If you skip preferences, you get unsubscribes and app-store one-stars. If you skip idempotency, a retry double-charges nothing — it double-sends a payment receipt, which is almost as bad.

How it works

Ingest. A product calls POST /notifications (or publishes to Kafka) with event_type, user_id, payload, and an idempotency key. The API writes a durable row and enqueues work. It does not wait for Gmail.
Resolve. A planner loads preferences (channel opt-in, quiet hours, locale) and templates. It may drop the event ("user muted this product") or split it (in-app always, push only if the app is backgrounded).
Dispatch. Channel workers pull jobs. Email talks to SES/SendGrid. Push talks to APNs/FCM. SMS talks to Twilio. In-app writes a row the client polls or receives over WebSocket.
Failure. Providers 429, timeout, or accept then bounce. You retry with backoff, honor provider retry-after, and park poison messages on a dead-letter queue. You store provider message IDs so a webhook ("delivered", "bounced") can update status.

Clients never talk to Twilio. APIs own auth and idempotency. Stores are typically: notification records (status machine), preference DB, template store, and a queue. Failure is the default path: the worker dies after SES accepted the mail — the next retry must see "already sent" and stop.

Priority queues help: security and money events jump the line; marketing waits.

A simple example

You ship an order. Checkout publishes order.shipped with key order_123_shipped. The planner: in-app inbox yes, email yes (user opted in), SMS no (quiet hours), push yes (mobile). Email worker sends once. The API retries because the HTTP client timed out after SES succeeded. Same idempotency key: worker sees status=sent and returns. The user gets one email, one push, one inbox row.

A flash sale fires a million price.drop events. Without per-user rate limits, you melt APNs and the user's lock screen. You cap "promo push" to one per hour and collapse duplicates in the planner.

Common mistakes

Calling the provider from the product service. Checkout now depends on Twilio latency and outages. Decouple.

No idempotency. At-least-once queues will send twice. Keys must cover "this event, this user, this channel."

One queue for everything. A stuck marketing blast should not delay "your 2FA code." Separate topics or priority.

Ignoring preferences and quiet hours. Technically delivered, product-wise a failure.

Treating in-app as an afterthought. It is the cheapest channel and the audit trail when email is delayed.

How this shows up in real systems

Slack, Discord, GitHub: mention fan-out plus digest email so you are not pinged per comment.
Stripe, banks: transactional email with strict templates and delivery webhooks.
Braze, Customer.io, OneSignal: the notification plane as a product — you still design the same pieces if you build in-house.

Layer 3's queues and Layer 2's retries are the spine. The new work is channel adapters and user policy.

Recap

Ingest events, plan deliveries, dispatch per channel — never send from the source product.
Idempotency, preferences, and priority decide whether the system is trusted.
Design for provider failure and worker crash as the normal case.

Chat is the next fan-out problem: not "notify later," but "deliver this message to everyone in the room now."

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 1 of 26

← Previous: Disaster Recovery → Next: Designing a Real-Time Chat System