πŸ’¬
βœ‰οΈ
πŸ“Έ
▢️

Loading…

Distributed systems: CAP, quorum, replication, and consensus

System Evolution

System Evolution

Step 2 of 14
πŸ”

Retries

Networks fail temporarily. Retrying is the first defense β€” but naive retries cause new problems.

Why the previous solution stopped being enough

Your monolith worked fine. Now your services call each other over a network. And networks sometimes drop packets.

Architecture Diagram

Request 1

β†’ fails (network blip)

Wait 100ms

exponential backoff

Retry 1

β†’ still fails

Wait 200ms

double the wait

Retry 2

β†’ success βœ“

In a monolith, calling a function always works or throws an exception you can handle. In distributed systems, a network request might get lost in transit, timeout, or return a server error that's entirely temporary.

A transient failure is one that would succeed if you tried again a moment later. A network blip, a server briefly overloaded, a pod restarting β€” these are not bugs. They're reality. Retrying solves them.

Naive retries are dangerous. If your service crashes under load, and every client immediately retries, you've just tripled your traffic load during the worst possible moment. This is a retry storm.

Exponential backoff with jitter solves this. After the first failure, wait 100ms. After the second, wait 200ms. After the third, 400ms. Add random jitter to spread retries across time so clients don't all retry simultaneously.

Set a retry limit. Retrying forever is not a recovery strategy β€” it's an infinite loop. Three attempts is a reasonable default for transient failures. After that, fail fast and let the caller decide what to do.

Tradeoffs

RecoveryTransient network failures become invisible to the user.
Retry stormsAll clients retrying at once multiplies load on an already-struggling service.
DuplicatesRetries can cause the same request to be processed multiple times.
Latency tailA failed + retried request takes longer. Watch your p99 latency under failure.

Engineering Edge Cases

β–Έ

Only retry on safe operations. Retrying a GET is fine. Retrying a payment without idempotency is not.

β–Έ

Circuit breakers complement retries: after enough failures, stop retrying entirely and fail fast.

What comes next

Retrying is essential β€” but it creates a new problem: what if the request was actually processed the first time, and the retry is a duplicate?