Distributed systems: CAP, quorum, replication, and consensus
System Evolution
Retries
Networks fail temporarily. Retrying is the first defense β but naive retries cause new problems.
Why the previous solution stopped being enough
Your monolith worked fine. Now your services call each other over a network. And networks sometimes drop packets.
Architecture Diagram
Request 1
β fails (network blip)
Wait 100ms
exponential backoff
Retry 1
β still fails
Wait 200ms
double the wait
Retry 2
β success β
In a monolith, calling a function always works or throws an exception you can handle. In distributed systems, a network request might get lost in transit, timeout, or return a server error that's entirely temporary.
A transient failure is one that would succeed if you tried again a moment later. A network blip, a server briefly overloaded, a pod restarting β these are not bugs. They're reality. Retrying solves them.
Naive retries are dangerous. If your service crashes under load, and every client immediately retries, you've just tripled your traffic load during the worst possible moment. This is a retry storm.
Exponential backoff with jitter solves this. After the first failure, wait 100ms. After the second, wait 200ms. After the third, 400ms. Add random jitter to spread retries across time so clients don't all retry simultaneously.
Set a retry limit. Retrying forever is not a recovery strategy β it's an infinite loop. Three attempts is a reasonable default for transient failures. After that, fail fast and let the caller decide what to do.
Tradeoffs
Engineering Edge Cases
Only retry on safe operations. Retrying a GET is fine. Retrying a payment without idempotency is not.
Circuit breakers complement retries: after enough failures, stop retrying entirely and fail fast.
What comes next
Retrying is essential β but it creates a new problem: what if the request was actually processed the first time, and the retry is a duplicate?
