Distributed systems: CAP, quorum, replication, and consensus
System Evolution
CAP Theorem
When the network splits, distributed systems must choose: consistent data or staying available.
Why the previous solution stopped being enough
Replication is running smoothly β until a network partition splits your cluster in two. Nodes on each side can still accept requests, but they can't communicate with each other. What does each side do?
Architecture Diagram
Consistency
every node returns same data
Availability
always responds
Partition Tolerance
survives network splits
CP systems
HBase, ZooKeeper
AP systems
Cassandra, DynamoDB
CA systems
not recommended
P is non-negotiable in distributed systems. You always choose between C and A.
The CAP Theorem, proven by Eric Brewer, states that a distributed system can only guarantee two of three properties simultaneously: Consistency, Availability, and Partition Tolerance.
Consistency means every read receives the most recent write. If you wrote 'balance = $500', every node immediately reflects $500. Availability means the system always responds β even if the response might be stale. Partition Tolerance means the system keeps running even when some network links between nodes fail.
In the real world, network partitions are unavoidable. Cables break, routers crash, cloud availability zones have outages. You cannot build a system that is both consistent and available during a partition β you must choose.
CP systems prioritize consistency over availability. During a partition, they refuse to serve reads rather than risk returning stale data. HBase, ZooKeeper, and Redis Cluster (with correct config) are examples. Banking and payment systems typically choose CP.
AP systems prioritize availability over consistency. During a partition, they continue serving requests but may return stale data. DynamoDB, Cassandra, and CouchDB are examples. Social feeds, shopping carts, and DNS are examples where AP is the right tradeoff.
PACELC extends CAP by noting that even without partitions, you trade latency for consistency: synchronous replication is more consistent but slower; async replication is faster but less consistent.
Tradeoffs
Engineering Edge Cases
CAP is often oversimplified. Many systems are tunable β you can pick consistency levels per operation.
Eventual consistency (AP) systems require application-level conflict resolution when diverged replicas reconcile.
What comes next
You understand the tradeoff. But how do systems actually enforce consistency across replicas when some nodes might be unreachable? That requires a concept called quorum.
