Distributed systems: CAP, quorum, replication, and consensus
System Evolution
Leader Election
Distributed systems need a single coordinator. Leader election chooses one β and replaces it when it fails.
Why the previous solution stopped being enough
Quorum works for per-operation coordination. But some tasks need a single node in charge: who decides which shard to assign to which server? Who processes the unprocessed queue items without duplicates? You need a leader.
Architecture Diagram
Node A
Leader β
Node B
Follower
Node C
Follower
Node D
Follower
Leader A fails
no heartbeat detected
Election triggered
nodes vote for new leader
Node B elected
majority vote wins
Leader election is the process by which distributed nodes agree on a single coordinator. That leader is responsible for centralized decisions: assigning work, managing cluster membership, or arbitrating conflicts.
The naive approach β nodes vote and whoever gets the most votes becomes leader β has a critical failure mode: split-brain. If a network partition divides the cluster in two, both halves might elect their own leader. Now you have two leaders making conflicting decisions.
Preventing split-brain requires quorum. A node can only become leader if it receives votes from a strict majority (>N/2) of nodes. In a 5-node cluster, you need 3 votes. If the cluster is split 2/3, only the larger partition can elect a leader.
ZooKeeper uses ZAB (ZooKeeper Atomic Broadcast) for leader election. etcd uses Raft. Both ensure that only one leader can exist at any time. When the leader fails (detected via heartbeat timeout), a new election begins.
Leader elections are designed to be infrequent β they're expensive and cause brief unavailability. The goal is a stable leader that serves for a long time, with fast failover when needed.
Tradeoffs
Engineering Edge Cases
Fencing tokens prevent split-brain writes: each leader gets an incrementing ID, and storage systems reject writes from older leader IDs.
The STONITH (Shoot The Other Node In The Head) pattern forcibly kills a suspected old leader before the new one takes over.
What comes next
Leader election picks a coordinator. But for critical distributed decisions β like committed a transaction across multiple services β even a leader isn't enough. You need all nodes to agree. That's consensus.
