Fencing Tokens and Leases

Pradeep P
2 days ago
3 min read

Series: Modern System Design · Layer 3 — Reliability

Layer 3 · Post 21 of 21

← Previous: Consensus: Paxos and Raft → Next: BASE

Layer 3 — Reliability · Post 105 of 119

A generation number or timed lease lets storage reject writes from a leader that does not know it was replaced.

What you'll learn

Why "the old primary paused" is more dangerous than "the old primary died"
Leases vs lock TTLs vs monotonic fencing tokens
Where ZooKeeper, etcd, and Kafka epochs show up

The idea in one minute

Election picked a new leader. The old one is not necessarily dead. It may be in a GC pause, a network partition that just healed one way, or a disk stall. It still has client connections. It writes.

A lease is a timed right to be leader: if you cannot renew, you must stop — even if you feel fine. A fencing token (epoch, generation, zid) is a number that only goes up. Storage (disk, object store, the next service) rejects any write with a stale token.

Old leader  token=7  --write-->  store  REJECT (current is 8)
New leader  token=8  --write-->  store  OK

Without this, Raft in the membership layer does not save you if the data plane still listens to the zombie.

Why it matters

This is the missing slide after every "we'll failover." Split-brain (Post 103) is the cluster view. Fencing is stopping I/O from the loser.

Interviews: they want monotonic epoch on every write, not only a Redis key lock=true with a 30s TTL. TTL locks are leases if and only if the holder stops on expiry and the resource checks the token. A lock that exists only in Redis while S3 still accepts the old writer's PUT is theater.

How it works

Lease. Leader has the lock until time T (plus some clock slack). It renews. If renewal fails, it abdicates. Unreliable clocks make this messy — hence bounded pause (do not GC for 2 minutes on a leader) and shared time (or etcd as the time source). Kubernetes leases for controllers are this.

Fencing token. When you acquire the lock, you get n+1. Every I/O carries n. The store stores last n. Old n is forbidden. ZooKeeper sequential znodes, etcd mod revision / lease IDs, Kafka producer epoch / partition leader epoch, EBS-style generation on attach: same shape.

STONITH (previous post) is fencing the machine. Tokens fence the writes when you cannot pull the plug fast enough (cloud APIs, paused VMs).

Safety vs liveness. If the new leader cannot fence the old (store down), do not take writes. That is ugly and correct. Taking writes then fencing is the window where you dual-write.

Not a mutex in the app only. The resource must enforce. The app can still try; the disk says no.

A simple example

A job runner: only one worker should write report.pdf to S3 for a given jobId.

Broken: Redis SET lock job:1 NX EX 30. Worker A holds it, pauses 35 s, Redis expires, worker B starts, A wakes and PUTs a half-finished file. Last writer wins; report is trash.

Fenced: lock service returns token=42. S3 (or a small metadata table) stores job:1 → 42. A's PUT includes If-token=42. After B acquires token=43, A's writes fail. A must check the error and stop, not retry blindly with the old token.

Kafka: a producer with an old epoch after a transaction abort / new leader is fenced; you must create a new producer. That is this post in the Kafka protocol.

Common mistakes

TTL lock without resource check. The TTL expired in Redis; the zombie did not notice.

Clock-only leases across machines. NTP step → two valid leases. Prefer a central lease service or tokens.

Incrementing the token in the client without the store. Both clients pick 8.

Long GC / stop-the-world on the leader. You designed a lease shorter than your pause. Either shorten pauses or lengthen leases and accept slower failover.

How this shows up in real systems

ZooKeeper / etcd locks with the version in the payload.
Kubernetes leader election leases on Lease objects.
Kafka ISR leader epoch; transactional producer fencing.
Cloud block storage attach generations; Ceph / GFS leases in papers.

Recap

A replaced leader may still write. Leases stop the process; tokens stop the store.
Enforce the token at the resource, not only in Redis.
Data-layer named patterns continue with BASE and outboxes.