Replication

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 3 — Reliability

Layer 3 · Post 9 of 14

← Previous: Backpressure → Next: Failover

Layer 3 — Reliability · Post 34 of 88

From a reliability angle, replication is how you survive machine loss. The questions become lag, failover, and whether readers see stale data.

What you'll learn

What replication lag does to failover (lost tail writes) and to reads (stale data)
Why "we have a replica" is not the same as a tested failover
How to talk about this in interviews without repeating the Layer 1 copy-the-bytes primer

The idea in one minute

Layer 1's Replication post is the building block: a primary takes writes, replicas copy a log, reads can fan out.

This post is the failure view. Copies are behind. Failover is lossy or slow. Readers can see yesterday.

Writes --> [ Primary ] --async log--> [ Replica ] --> reads | ^ x crash | +---- promote? last 200 ms of commits maybe never got here

You buy survival. You pay lag, stale reads, and failover risk. That trade is the reliability design.

Why it matters

A single node dying should not delete the company. The outage report is rarely "we forgot replicas." It is lag, lost tail writes on promote, DNS still on the dead primary, or a replica showing an empty cart after checkout.

Interviews: they already assume you know primary/replica. They want RPO (how much data you can lose) and RTO (how fast you are back), and whether reads are allowed to be stale.

How it works

Lag is the reliability metric. Bytes or seconds behind. Async replication (default Postgres streaming, many MySQL replicas, Redis replicas) means the primary can ack a commit the replica has not seen. If you fail over now, those commits are gone unless you had synchronous replication (or a quorum — later posts).

Stale reads. Load-balanced SELECTs hit replicas. After a write, a GET may miss it. User thinks save failed. Reliability here is a consistency choice (Post 39): read-your-writes via primary, session sticky, or "sync replica before read."

Failover is a separate mechanism. Replication only means another copy exists. Something must stop the old primary (fencing), promote a replica, and point clients (VIP, DNS, PgBouncer, RDS endpoint). If two primaries accept writes, you get split-brain — next two posts.

Sync vs async is an SLO choice. Sync: smaller RPO, higher write latency, writes stall if the replica is unreachable. Async: fast acks, RPO equals lag at crash time. Multi-AZ RDS often syncs in-region; cross-region replicas are usually async. Replicas also copy DELETEs — backups remain a different tool.

A simple example

Checkout writes orders to Postgres primary in us-east-1. Replica in us-east-1b, lag usually 30 ms. You fail over in 60 seconds via RDS Multi-AZ. Users see a brief error; orders that got an ack are there.

An async replica in eu-west-1 for reads can be 2 seconds behind: the user sees "payment pending" after a success screen. Stick that session to the primary after POST, or accept the glitch. If you fail over to Europe, know the lag — twelve minutes behind is twelve minutes of orders gone unless you wait or refuse to promote.

Common mistakes

Treating replica count as backups. Replication copies mistakes live.

Never measuring lag. Failover theater.

Reading replicas for "read-your-writes" UX (profile update, cart) with no sticky path.

Promoting without fencing the old primary. It comes back and accepts writes. Split-brain. Failover is next.

Sharing the HA replica with a 2-hour analytics query. You created lag, then fail over onto a stale copy.

How this shows up in real systems

Amazon RDS / Aurora Multi-AZ, Cloud SQL HA: sync-ish standby, automatic failover, lag metrics.
Postgres streaming replication, Patroni; MySQL binlog replicas; Redis replica + Sentinel.
Kafka ISR: a replica not in-sync is not a safe failover target.
DynamoDB / Aurora global / Cosmos: replication as a product with explicit consistency knobs (Post 39).

Recap

Replication is survival plus lag. Failover can lose the tail; reads can be stale.
Measure lag; separate copying from promoting and pointing.
Layer 1 was how copies work. Here, copies fail. Next: failover without two brains.