Failover

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 3 — Reliability

Layer 3 · Post 10 of 14

← Previous: Replication → Next: Leader Election

Layer 3 — Reliability · Post 35 of 88

Failover is the process of moving traffic from a sick primary to a healthy standby. The hard part is doing it quickly without splitting the brain.

What you'll learn

The three steps: detect, fence, redirect — missing any one is a new outage
Split-brain: two primaries, divergent data, and why fencing exists
Automatic vs human-gated failover, and what RTO/RPO actually mean here

The idea in one minute

Failover is moving the live role (primary, leader, active VIP) from a sick node to a healthy one.

Replication (previous post) copied the data. Failover is the cut.

Detect: health check fails / no heartbeat
Fence: old primary must not accept writes (kill, revoke lease, STONITH)
Promote standby + point clients (DNS, VIP, connection proxy)

Skip fencing and both nodes think they are primary. That is split-brain. You will merge data with a prayer.

Why it matters

Users do not care that a replica existed. They care that checkout continued. The reliability numbers:

RTO: how long until a healthy primary is serving.
RPO: how much acknowledged data you can lose (lag at cut time).

Interviews: draw Multi-AZ and then explain who decides, how clients find the new node, and what happens if the old one was just a network blip. "We fail over" without fencing is how you get two ledgers.

How it works

Detect. Health checks and leases. Too aggressive: you fail over on a GC pause (flapping). Too slow: you sit on a dead primary for minutes. Heartbeats should be shorter than your RTO, with a timeout that is not a single packet loss.

Fence. The old primary must lose the right to write. Techniques: take the disk/VIP away, shoot the node (STONITH), expire a ZooKeeper/etcd lease so it steps down, RDS fencing internally. The rule: never promote until the old writer is fenced (or you accepted split-brain).

Redirect. Clients need a stable name: RDS endpoint, Kubernetes Service, Envoy cluster, elastic IP, DNS. DNS failover is slow unless TTLs are tiny and resolvers honor them (they often do not). Prefer a proxy or VIP that you flip.

Automatic vs manual. Automatic is the only way to hit a 30-second RTO at 3 a.m. It also fails over on false positives. Many teams auto-fail within a zone (RDS Multi-AZ) and human-gate cross-region (lag, legal, "did we really lose the region?").

Failback is another failover. Do it on purpose, with lag caught up, not when the old node boots and grabs the VIP.

Stateless services "fail over" with a load balancer (Post 4). This post is about state: one writer.

A simple example

Postgres primary dies. Patroni/etcd lease expires. A replica wins election (next post), is promoted, VIP moves. Apps through PgBouncer reconnect. RTO ~30 s. RPO ~0 if you used a sync standby; seconds if async.

Bad version: a cron pings the primary, times out once, promotes the replica. Primary was a network partition. It still has the VIP on its side of the split. Two writers. Overnight, two "unique" order ids. Monday is a forensic restore.

Common mistakes

Failover = DNS only, TTL 60 s. Clients cache for five minutes. Use a VIP or proxy.

No fencing. The original sin.

Failing over on one failed health check. Flap until the cluster is mush.

Never rehearsing. The replica is a year behind on extensions, or the connection string is hardcoded.

Failing over the database but not the cache. Redis still has old session truth; users bounce.

Treating load balancer target draining as database failover. Different problem (stateless vs one writer).

How this shows up in real systems

RDS Multi-AZ, Aurora, Cloud SQL HA: managed detect + fence + endpoint flip.
Redis Sentinel / Redis Cluster, Patroni, MongoDB replica sets.
Kubernetes: kube-proxy / Services for stateless; operators for stateful failover.
Route 53 health checks: coarse, DNS-bound; fine for whole-region websites, clumsy for Postgres.
Envoy / HAProxy as the pointer clients actually use.

Recap

Failover is detect, fence, redirect — not "promote a replica."
Split-brain is two writers; fencing is how you prevent it.
Practice it; know RTO/RPO. Next: who is allowed to be the writer — leader election.