Distributed systems: CAP, quorum, replication, and consensus
System Evolution
Consensus
Distributed agreement is the hardest problem in systems. Consensus algorithms solve it under failures.
Why the previous solution stopped being enough
Leader election works for cluster coordination. But your distributed transaction system needs all services to agree on whether a transaction committed β even if some nodes crash halfway through.
Architecture Diagram
Proposer
Acceptor A
accepts
Acceptor B
accepts
Acceptor C
accepts
Majority accepted β value committed
Paxos / Raft: even if some nodes fail, the system agrees
Raft
leader-based, simpler
Paxos
flexible, complex
Consensus is the problem of getting multiple nodes to agree on a single value, even in the presence of failures. This sounds simple β just have them vote β but failure modes make it subtle.
Consider: Node A sends a value to 5 nodes. Nodes 1, 2, and 3 receive it and confirm. Node 4 receives it but crashes before responding. Node 5 never receives it due to a network drop. Has the value been committed? What does Node 5 do if it later recovers?
Paxos, invented by Leslie Lamport in 1989, was the first rigorous solution. In Paxos, a Proposer sends a value to Acceptors. Acceptors commit once a majority promises. Even if the Proposer crashes, the already-committed majority ensures the value persists.
Raft, published in 2014, solves the same problem but is designed to be understandable. In Raft, a leader receives all writes, replicates them to followers, and commits once a majority acknowledges. Leader election is a separate clearly-defined phase.
Consensus powers the most critical infrastructure: distributed databases (etcd in Kubernetes, CockroachDB), coordination services (ZooKeeper), and distributed locks. Any time you need a guarantee that a group of nodes agree on something β configuration, leader identity, transaction commit β you're using consensus.
Tradeoffs
Engineering Edge Cases
Multi-Paxos pipelines multiple operations, amortizing the two-phase cost over many writes.
Byzantine fault tolerance (BFT) handles malicious nodes, not just crashed ones β used in blockchains, not typical data systems.
What comes next
Your distributed system can now agree on things reliably. But data is partitioned across many shards, and not all shards receive equal traffic. Some shards are crushing under load while others sit idle.
