πŸ’¬
βœ‰οΈ
πŸ“Έ
▢️

Loading…

Distributed systems: CAP, quorum, replication, and consensus

System Evolution

System Evolution

Step 10 of 14
πŸ‘‘

Leader Election

Distributed systems need a single coordinator. Leader election chooses one β€” and replaces it when it fails.

Why the previous solution stopped being enough

Quorum works for per-operation coordination. But some tasks need a single node in charge: who decides which shard to assign to which server? Who processes the unprocessed queue items without duplicates? You need a leader.

Architecture Diagram

Node A

Leader β˜…

β–Ό heartbeats

Node B

Follower

Node C

Follower

Node D

Follower

Leader A fails

no heartbeat detected

Election triggered

nodes vote for new leader

Node B elected

majority vote wins

Leader election is the process by which distributed nodes agree on a single coordinator. That leader is responsible for centralized decisions: assigning work, managing cluster membership, or arbitrating conflicts.

The naive approach β€” nodes vote and whoever gets the most votes becomes leader β€” has a critical failure mode: split-brain. If a network partition divides the cluster in two, both halves might elect their own leader. Now you have two leaders making conflicting decisions.

Preventing split-brain requires quorum. A node can only become leader if it receives votes from a strict majority (>N/2) of nodes. In a 5-node cluster, you need 3 votes. If the cluster is split 2/3, only the larger partition can elect a leader.

ZooKeeper uses ZAB (ZooKeeper Atomic Broadcast) for leader election. etcd uses Raft. Both ensure that only one leader can exist at any time. When the leader fails (detected via heartbeat timeout), a new election begins.

Leader elections are designed to be infrequent β€” they're expensive and cause brief unavailability. The goal is a stable leader that serves for a long time, with fast failover when needed.

Tradeoffs

Single coordinatorCentralized decisions avoid the complexity of distributed coordination for every operation.
Election gapDuring leader election, some operations must wait. Unavailability window of seconds to tens of seconds.
Split-brain riskPoorly designed election can create two leaders β€” a catastrophic state requiring manual intervention.
Leader hotspotAll coordination traffic goes through the leader β€” it can become a bottleneck.

Engineering Edge Cases

β–Έ

Fencing tokens prevent split-brain writes: each leader gets an incrementing ID, and storage systems reject writes from older leader IDs.

β–Έ

The STONITH (Shoot The Other Node In The Head) pattern forcibly kills a suspected old leader before the new one takes over.

What comes next

Leader election picks a coordinator. But for critical distributed decisions β€” like committed a transaction across multiple services β€” even a leader isn't enough. You need all nodes to agree. That's consensus.