Leader Election

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 3 — Reliability

Layer 3 · Post 11 of 14

← Previous: Failover → Next: Quorum

Layer 3 — Reliability · Post 36 of 88

Leader election picks one node to make decisions the others will follow, so a cluster does not argue with itself.

What you'll learn

Why "exactly one writer/coordinator" is a cluster problem, not a config file
Leases, fencing tokens, and how a dead leader is replaced without two leaders
Where Raft, etcd, and ZooKeeper show up so you do not invent this in the app

The idea in one minute

Leader election is how a set of nodes agrees: this one is the leader (primary, controller, scheduler). Everyone else follows or stands by.

Failover (previous post) uses an election (or a human) to pick the next writer. Election is the algorithm; failover is the cutover.

Nodes A B C vote / lease | v A is leader --> A takes writes, B and C replicate A silent --> new election --> B is leader A must not keep writing (lease expired / fenced)

If two nodes believe they won, you have split-brain with a democratic origin story.

Why it matters

Anything that must be singular — one primary database, one Kubernetes controller reconcile loop, one Kafka partition leader, one cron that sends invoices — needs a leader or a lock. "We'll start only one pod" is not election; it is hope (deploys, partitions, "oops, two replicas").

Interviews: they want leases, not if hostname == db01. They want you to name etcd / ZooKeeper / Raft instead of a home-grown UPDATE leaders SET ... without fencing.

How it works

Majority vote (Raft, etcd, ZooKeeper). Odd-sized cluster (3, 5). A candidate needs a quorum (next post) of votes. A partition that holds a minority cannot elect a leader, so it cannot write. That is the point: you chose consistency over "always someone in charge."

Lease. The leader must renew. Miss the deadline → others may elect. The old leader must stop on expiry, even if it is only partitioned from the voters but still serving clients. That is fencing. A fencing token (monotonic epoch): storage rejects writes from a stale epoch. ZooKeeper sequential znodes and etcd compare-and-swap are this idea.

Not a heartbeat to a buddy. A and B ping each other, the link dies, both promote. Classic split-brain. You need a shared source of truth (quorum, or a STONITH device), not pairwise optimism.

App-level: don't run Raft in your checkout service. Use:

the database's own election (Postgres + Patroni on etcd),
or a lock in etcd/ZooKeeper/Redis (Redis locks are easy to get wrong — prefer fencing),
or a managed primary (RDS).

Leader is for coordination. It is not required for every read. Replicas still serve reads (and can be stale — Post 34).

A simple example

Three-node etcd elects a leader. Kubernetes API servers use etcd. The controller manager holds a lease so only one replica runs Deployment reconcile. If that pod dies, the lease expires, another pod becomes leader, reconcile continues. Two controller managers without a lease duplicate ReplicaSets.

Kafka: each partition has a leader broker. Followers fetch. If the leader dies, the controller (itself elected) picks a new leader from the ISR. Producers that ignore epoch can hit the old leader — brokers fence by epoch.

Common mistakes

replicas: 2 and a boolean isLeader in Redis without expiry. Network split, two leaders.

Election among two nodes. 1-1 tie, or one node partitioned has no majority. Use 3.

Ignoring the old leader. Election without fencing.

Making the leader do all the work. Election is for decisions (writes, schedule). Fan out execution. A leader that is also the only worker is a bottleneck you just elected.

Home-grown Raft. Use etcd, Consul, ZooKeeper, or the database vendor.

How this shows up in real systems

etcd, ZooKeeper, Consul: the lock/election services.
Kubernetes leases, Kafka controller and partition leaders, Elasticsearch elected master.
Raft inside etcd, Consul, MongoDB replica sets, Cockroach, TiKV.
Redis Sentinel elects a primary; Patroni uses etcd/Consul for Postgres.

Recap

Election picks one leader so the cluster does not argue.
Leases + fencing + quorum beat pinging a friend.
Do not invent it; use etcd/Raft/vendor HA. Next: the vote count that makes this safe — quorum.