Distributed systems: CAP, quorum, replication, and consensus
System Evolution
Hot Keys & Hotspotting
Distributed systems assume traffic is spread evenly. Reality disagrees β and the hot shard crashes.
Why the previous solution stopped being enough
Consensus and coordination are solid. But your sharded database cannot survive viral traffic. When BeyoncΓ© drops an album, 50 million users hit the same artist ID within minutes β all routed to shard 3.
Architecture Diagram
Shard 1
normal
Shard 2
normal
Shard 3
π₯ hot
Shard 4
normal
Shard 5
normal
Celebrity post: 100M views in 1 hour
All traffic β Shard 3 β CPU 100% β crashes
When you partition data across shards (by user ID, artist ID, product ID), you assume traffic will distribute roughly evenly. For most data, this is true. But some keys have disproportionate traffic β these are hot keys.
The celebrity problem is the canonical example: a social platform shards user data by user ID. Most users get 1,000 requests/day. But Cristiano Ronaldo gets 1 billion. His shard receives 1 million times more traffic. The math doesn't work.
The same issue appears with viral content (a tweet goes viral β one partition is overloaded), event-driven spikes (an e-commerce flash sale β one product's shard), and time-series data (today's date key receives all current writes).
Solutions for hot keys include: read replicas for the hot key specifically, adding a suffix to shard the hot key across multiple partitions (user:12345:0 through user:12345:9 β spread across 10 shards), caching the hot key aggressively at the application layer, and using async fan-out to distribute the write impact.
Detection is critical. Set up partition-level metrics in your database β reads/sec, CPU, latency per shard. An imbalanced histogram tells you immediately which partition is hot before it causes an outage.
Tradeoffs
Engineering Edge Cases
Time-based hotspots are predictable: always add a random suffix to current-time bucket keys to pre-distribute them.
Consistent hashing reduces key migration during rebalancing but doesn't prevent hotspots β those require separate treatment.
What comes next
You've handled the hot keys. But over time, as nodes are added and removed, data distribution becomes uneven across all partitions β not just the hot ones. The system needs to redistribute data.
