πŸ’¬
βœ‰οΈ
πŸ“Έ
▢️

Loading…

Distributed systems: CAP, quorum, replication, and consensus

System Evolution

System Evolution

Step 7 of 14
πŸ“‹

Replication

One database can't serve all reads. Replication spreads the load across copies of your data.

Why the previous solution stopped being enough

Service discovery is working well. But your database is overwhelmed β€” 95% of queries are reads. A single primary DB is at 80% CPU serving the read traffic alone, with no headroom for writes.

Architecture Diagram

Client Writes

Primary DB

accepts all writes

async copy

Replica 1

read only

async copy

Replica 2

read only

async copy

Replica 3

read only

Stale reads possible

3Γ— read throughput

Replication means maintaining copies of your data on multiple servers. The most common pattern is primary-replica (formerly called master-slave): one node accepts all writes, and those writes are asynchronously or synchronously copied to N replica nodes. Reads can be served from any replica.

For a read-heavy system (social feeds, product catalogs, analytics dashboards), even 3 replicas can give you 3Γ— read throughput without touching your primary. This is the most impactful scaling step most teams skip in favor of more complex solutions.

Replication lag is the delay between a write landing on the primary and that write being visible on a replica. For async replication, this is usually milliseconds. But during high write load or network congestion, it can grow to seconds β€” or more.

Stale reads are a consequence. If you write to the primary and immediately read from a replica, you might get the old value. This is acceptable for many use cases (feed content, product details) but not for others (your own account balance after a transfer).

Synchronous replication eliminates stale reads: the write is not acknowledged until at least one replica confirms it. But it adds write latency. Most production systems use async replication for performance and tolerate eventual consistency for most read paths.

Tradeoffs

Read scalabilityLinear read scaling: 3 replicas β‰ˆ 3Γ— your original read capacity.
Stale readsAsync replication means replicas can be behind. Reads may return outdated data.
Write bottleneck remainsAll writes still go to the primary. Replication doesn't solve write scaling.
Replication lagUnder heavy write load, lag can grow β€” replicas fall further behind.

Engineering Edge Cases

β–Έ

Read-your-own-writes consistency: after writing, route the immediate read back to the primary to avoid seeing stale data.

β–Έ

Replica failover: when the primary fails, a replica must be promoted. This requires careful coordination to avoid data loss.

What comes next

Replication works well. But you now have multiple copies of data that can diverge. What happens when a network partition splits your nodes? Which version of the data is correct?