πŸ’¬
βœ‰οΈ
πŸ“Έ
▢️

Loading…

Distributed systems: CAP, quorum, replication, and consensus

System Evolution

System Evolution

Step 12 of 14
πŸ”₯

Hot Keys & Hotspotting

Distributed systems assume traffic is spread evenly. Reality disagrees β€” and the hot shard crashes.

Why the previous solution stopped being enough

Consensus and coordination are solid. But your sharded database cannot survive viral traffic. When BeyoncΓ© drops an album, 50 million users hit the same artist ID within minutes β€” all routed to shard 3.

Architecture Diagram

Shard 1

normal

Shard 2

normal

Shard 3

πŸ”₯ hot

Shard 4

normal

Shard 5

normal

Celebrity post: 100M views in 1 hour

All traffic β†’ Shard 3 β†’ CPU 100% β†’ crashes

When you partition data across shards (by user ID, artist ID, product ID), you assume traffic will distribute roughly evenly. For most data, this is true. But some keys have disproportionate traffic β€” these are hot keys.

The celebrity problem is the canonical example: a social platform shards user data by user ID. Most users get 1,000 requests/day. But Cristiano Ronaldo gets 1 billion. His shard receives 1 million times more traffic. The math doesn't work.

The same issue appears with viral content (a tweet goes viral β†’ one partition is overloaded), event-driven spikes (an e-commerce flash sale β†’ one product's shard), and time-series data (today's date key receives all current writes).

Solutions for hot keys include: read replicas for the hot key specifically, adding a suffix to shard the hot key across multiple partitions (user:12345:0 through user:12345:9 β€” spread across 10 shards), caching the hot key aggressively at the application layer, and using async fan-out to distribute the write impact.

Detection is critical. Set up partition-level metrics in your database β€” reads/sec, CPU, latency per shard. An imbalanced histogram tells you immediately which partition is hot before it causes an outage.

Tradeoffs

Even distributionWell-designed sharding gives linear scalability. Hotspots eliminate it.
Cache dependencyHot keys often require in-front caching β€” Memcached, Redis β€” to survive spikes.
Complex sharding logicSplitting hot keys across sub-shards requires application awareness of the scheme.
UnpredictabilityYou can't always predict which keys will go hot β€” defensive design matters.

Engineering Edge Cases

β–Έ

Time-based hotspots are predictable: always add a random suffix to current-time bucket keys to pre-distribute them.

β–Έ

Consistent hashing reduces key migration during rebalancing but doesn't prevent hotspots β€” those require separate treatment.

What comes next

You've handled the hot keys. But over time, as nodes are added and removed, data distribution becomes uneven across all partitions β€” not just the hot ones. The system needs to redistribute data.