πŸ’¬
βœ‰οΈ
πŸ“Έ
▢️

Loading…

Distributed systems: CAP, quorum, replication, and consensus

System Evolution

System Evolution

Step 11 of 14
🀝

Consensus

Distributed agreement is the hardest problem in systems. Consensus algorithms solve it under failures.

Why the previous solution stopped being enough

Leader election works for cluster coordination. But your distributed transaction system needs all services to agree on whether a transaction committed β€” even if some nodes crash halfway through.

Architecture Diagram

Proposer

Propose: x=7

Acceptor A

accepts

Acceptor B

accepts

Acceptor C

accepts

Majority accepted β†’ value committed

Paxos / Raft: even if some nodes fail, the system agrees

Raft

leader-based, simpler

Paxos

flexible, complex

Consensus is the problem of getting multiple nodes to agree on a single value, even in the presence of failures. This sounds simple β€” just have them vote β€” but failure modes make it subtle.

Consider: Node A sends a value to 5 nodes. Nodes 1, 2, and 3 receive it and confirm. Node 4 receives it but crashes before responding. Node 5 never receives it due to a network drop. Has the value been committed? What does Node 5 do if it later recovers?

Paxos, invented by Leslie Lamport in 1989, was the first rigorous solution. In Paxos, a Proposer sends a value to Acceptors. Acceptors commit once a majority promises. Even if the Proposer crashes, the already-committed majority ensures the value persists.

Raft, published in 2014, solves the same problem but is designed to be understandable. In Raft, a leader receives all writes, replicates them to followers, and commits once a majority acknowledges. Leader election is a separate clearly-defined phase.

Consensus powers the most critical infrastructure: distributed databases (etcd in Kubernetes, CockroachDB), coordination services (ZooKeeper), and distributed locks. Any time you need a guarantee that a group of nodes agree on something β€” configuration, leader identity, transaction commit β€” you're using consensus.

Tradeoffs

Strong guaranteesCommitted values are safe even if f nodes fail (where f < N/2).
LatencyRequires a round-trip to a majority before acknowledging a write. Unavoidable overhead.
Throughput ceilingAll writes go through a single leader. Consensus is inherently serialized.
ComplexityPaxos edge cases fill research papers. Even Raft has subtle correctness conditions.

Engineering Edge Cases

β–Έ

Multi-Paxos pipelines multiple operations, amortizing the two-phase cost over many writes.

β–Έ

Byzantine fault tolerance (BFT) handles malicious nodes, not just crashed ones β€” used in blockchains, not typical data systems.

What comes next

Your distributed system can now agree on things reliably. But data is partitioned across many shards, and not all shards receive equal traffic. Some shards are crushing under load while others sit idle.