Designing a Real-Time Chat System

Pradeep P
3 days ago
4 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 2 of 26

← Previous: Designing a Notification System → Next: Designing a Search System

Layer 6 — Modern systems · Post 64 of 88

A chat system must fan messages out quickly, keep conversation history, handle presence, and stay correct when users are on several devices.

What you'll learn

Why the write path (persist, then fan-out) and the read path (history + live socket) are different systems glued together
How connection servers, a message store, and pub/sub get a line to every device without polling the database
What "ordering" and "unread" actually mean when phones go offline and come back

The idea in one minute

Chat is durable messaging with a live overlay. The user types. You assign an ID, persist the message, then push it to everyone currently connected to that room. History is a query. Presence is a side channel ("typing", "online").

Phone / Web / Desktop | v [ Gateway: WebSocket / HTTP ] ---- persist ----> [ Message store ] | | | subscribe | v v [ Connection servers ] <---- pub/sub (room N) ---- [ Fan-out bus ] | drop? client reconnects, fetches catch-up by last_id

If you only persist, it feels like email. If you only push, a refresh loses the conversation. You need both.

Why it matters

This is a standard interview because it forces you to talk about fan-out, consistency, and multi-device. Slack is not "a WebSocket." WhatsApp is not "a database." The question is how those pieces share a message ID and a cursor.

Wrong ordering in a 1:1 chat is annoying. Wrong unread counts in a workplace tool looks like dropped work.

How it works

Clients open a long-lived connection to a gateway (WebSocket, or SSE for receive-only). The gateway authenticates, then maps user_id → this TCP connection. Users on three devices have three connections.
Send API. POST /rooms/{id}/messages (or the same over the socket) validates membership, writes the message to the store (often a wide-row or partition-per-room log: room_id, seq or snowflake_id, body, sender, timestamp). The write is the source of truth.
Fan-out. After commit, publish {room_id, message_id} on a bus (Redis Pub/Sub, Kafka, NATS). Connection servers that hold sockets for members of that room push the payload. For 1:1 this is tiny. For a 50k-member channel you do not push to 50k sockets from one box — you shard connection servers and publish to a room topic.
History. New clients or reconnects call GET /rooms/{id}/messages?after=cursor. They do not trust the socket alone.
Presence and typing are ephemeral: heartbeat in Redis with a short TTL. Do not write "Alice is typing" to Postgres on every keystroke.
Failure. Gateway dies: clients reconnect, resume by cursor. Store is down: you fail the send (better than a ghost message). Bus is down: persist succeeded, live push delayed — reconnect catch-up still works if history is correct.

Unread is a per-user cursor (last_read_seq) in a separate store, not a scan of the message table.

Group chat vs 1:1 is the same pipeline with different fan-out size. For huge rooms, some products persist once and let clients poll or use a channel subscription, not a per-user inbox copy.

A simple example

You send "on my way" from your phone. The API writes msg_9f3 with seq=1842 in room dm_ann_you. Pub/sub hits the web gateway where Ann's laptop is connected; her UI appends the row. Your tablet was asleep. It reconnects, asks after=1841, gets msg_9f3, and the thread matches. Ann's unread badge on mobile increments until that device's cursor advances.

If the phone retries the POST because the response was lost, the client idempotency key (or server-side dedupe on client-generated ID) prevents two "on my way" lines.

Common mistakes

Fan-out before persist. Recipients see a message that a crash then loses. Persist first, or make the push explicitly "unconfirmed."

Polling the message table for every online user. It will not scale. Connections + pub/sub exist for a reason.

One global sequence for the whole product. You want per-room ordering. Global clocks lie; room sequences or sortable IDs do not have to mean "happened-before across rooms."

Storing presence in the primary DB. You will melt it. Use a cache with TTLs.

Ignoring multi-device. "Delivered" on one phone is not "read" on the laptop unless you define those receipts.

How this shows up in real systems

WhatsApp / Signal: end-to-end encryption adds device sessions and cannot store plaintext the same way; the fan-out and catch-up ideas remain.
Slack / Discord: channel membership, huge rooms, search as a separate index (next posts).
Intercom / support chat: same pattern, plus routing to an agent queue.

Layer 2 connections and Layer 3 pub/sub do the realtime. The interview win is cursor-based catch-up and persist-then-fan-out.

Recap

Persist the message, then fan it out to connection servers; history is a cursor query.
Presence is ephemeral. Unread is a per-user pointer, not a full scan.
Design reconnect and retries as first-class — sockets drop constantly.

Once messages exist as documents, people will want to find them. That is search.

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 2 of 26

← Previous: Designing a Notification System → Next: Designing a Search System