Split-Brain

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 3 — Reliability

Layer 3 · Post 19 of 21

← Previous: PACELC → Next: Consensus: Paxos and Raft

Layer 3 — Reliability · Post 103 of 119

A network cut can leave two leaders, two writers, and diverging data until you fence one side.

What you'll learn

How a partition plus optimistic failover creates two primaries
Why majority quorum prevents it, and what STONITH/fencing do when you cannot wait
How to talk about this in interviews without only saying "we use Raft"

The idea in one minute

Split-brain: the cluster cuts into two (or more) groups that cannot talk, and more than one group believes it is in charge.

Each side takes writes. You now have two histories. When the cable returns, there is no automatic "true" copy — you merge, pick a winner, or restore from backup.

{ Primary A, clients } ~~~~ cut ~~~~ { Replica B, more clients }
A still accepts writes                     B promotes itself (timeout)
               two sources of truth

Failover (Post 35) without quorum (Post 37) or fencing (Post 105) is how you buy this on purpose.

Why it matters

This is the named incident behind "we failed over and corrupted the database." CAP's CP choice is often to avoid split-brain: the minority must not write. AP systems allow divergence and name the merge.

Interviews: they want majority of 3, not two nodes pinging each other. Two-node clusters are famous split-brain machines: when the link dies, both think the other is dead.

How it works

How you get there.

Heartbeat between two nodes, no third witness.
Load balancer still sending traffic to both after a partition.
replicas: 2 on a controller without a lease.
Async replica promoted while the old primary is still reachable by some clients (the classic "I failed over but DNS/clients didn't").

How you avoid two writers.

Quorum: only a majority elects / accepts writes. The minority goes read-only or errors (CP).
Witness / arbiter: a tiny third voter (Mongo arbiter, some Pacemaker setups) so 2-data-node clusters can still have majority. The arbiter must not become a silent SPOF in another failure mode — know the docs.
Fencing (STONITH): the new primary kills the old one's access (power off, revoke SCSI, block at the storage layer) before taking writes. "Shoot the other node in the head" so it cannot write even if it feels healthy.
Fencing tokens: storage rejects writes from the stale generation (next posts).

How you recover if it already happened. Pick a source of truth (newer LSN, more rows — dangerous), rewind the loser, or rebuild. There is no pretty algorithm that always preserves both sides' writes. That is why we try not to get here.

AP split-brain is sometimes intentional (multi-primary, CRDTs). Then it is not an accident; it is conflict resolution. Still name it.

A simple example

Two Postgres boxes, Pacemaker, heartbeat cable. Cable unplugged. Both promote. App DNS round-robins. Orders 1001–1100 on A, 1001–1080 different rows on B. Cable back: replication is confused; unique keys collide.

A 3-node etcd / Patroni cluster: the side with one node cannot win the election. Clients to the minority get errors. Painful, not divergent. That is the CP split-brain prevention.

Common mistakes

Two-node HA as a checkbox. Add a witness or accept that a partition will dual-promote.

Failover runbook that promotes first and fences later. There is a window. Fence then promote, or use a protocol that does it atomically.

"We'll merge with last-write-wins." On a ledger, LWW is data loss. Use it only where the product allows.

Ignoring clients. Split-brain at the DB plus sticky sessions to both primaries is twice the damage. Fencing must include I/O path, not only the peer heartbeat.

How this shows up in real systems

Elasticsearch: historic split-brain; minimum_master_nodes / today's voting config — learn the current setting.
Redis Sentinel / cluster: majority of sentinels / masters; mis-size and you dual-master.
Kubernetes: two control planes without etcd quorum — two apiservers with different realities.
DR drills: promoting the DR site while production is still live is split-brain you scheduled.

Recap

Split-brain = two active primaries (or two truth sets) after a cut.
Prevent with quorum + fencing; recover with choose-and-rewind, not hope.
Next: the algorithms that elect one history — Paxos and Raft.