Distributed systems: CAP, quorum, replication, and consensus
System Evolution
Rebalancing
Adding or removing nodes creates imbalance. Rebalancing redistributes data β at a cost.
Why the previous solution stopped being enough
Hot keys are handled. But you've added 4 new database nodes to increase capacity. The old nodes are at 85% utilization while the new nodes sit empty. The cluster is unbalanced.
Architecture Diagram
Before
Node A
78%
Node B
82%
Node C
12%
rebalance
After
Node A
51%
Node B
49%
Node C
50%
During migration: some reads may hit wrong node. Temporary inconsistency.
As distributed systems grow β adding nodes for capacity, removing nodes that are failing, or replacing hardware β the distribution of data across nodes becomes uneven. Rebalancing moves partitions between nodes to restore equilibrium.
Consistent hashing minimizes the amount of data moved during rebalancing. Instead of remapping all keys when a new node is added, only the keys that 'belong' to the new node's range are moved. For N nodes and K keys, adding a node moves only K/N keys β a fraction of the total.
Rebalancing is expensive. Moving data means reading from one node and writing to another over the network. During a large rebalance, network bandwidth is consumed, disk I/O spikes, and β if not throttled β query latency increases for production traffic.
Most production systems implement throttled rebalancing: moving data at a rate that keeps production acceptable, even if the rebalance takes hours instead of minutes. Cassandra's nodetool repair and Kafka's partition reassignment both work this way.
Temporary inconsistency is a reality during rebalancing. A key that's being migrated might be readable on the old node but not yet written to the new node. Systems handle this with forwarding: the old node forwards requests to the new node during migration.
Tradeoffs
Engineering Edge Cases
Never add too many nodes at once. A large-scale rebalance of 30% of data simultaneously can saturate the network and cause outages.
Verify rebalancing completion before decommissioning old nodes. Premature decommission causes data loss.
What comes next
Storage is balanced. But your services are still tightly coupled β Service A blocks waiting for Service B's response. Under load, that synchronous chain multiplies latency. You need to decouple them.
