πŸ’¬
βœ‰οΈ
πŸ“Έ
▢️

Loading…

Distributed systems: CAP, quorum, replication, and consensus

System Evolution

System Evolution

Step 3 of 14
πŸ”‘

Idempotency

Retries create duplicates. Idempotency ensures that repeating an operation is safe.

Why the previous solution stopped being enough

Retries saved you from transient failures. But now your payment service is processing the same charge twice β€” once from the original request, once from the retry.

Architecture Diagram

Client

3Γ— retries

Payment Service

Idempotency Key

txn_a8f91bz

Dedup Store

seen before?

Process once

Return cached result

An idempotent operation is one where doing it multiple times has the same result as doing it once. DELETE a file β€” if you issue that command ten times, you still end up with no file. That's idempotent. Transfer $100 β€” if you issue that ten times, you've now sent $1000. That is not idempotent.

The classic horror story: a user clicks 'Pay'. The request goes through, the payment is processed, but the response gets lost in transit. The client times out and retries. The server processes the payment again. The user has been charged twice.

The solution is an idempotency key β€” a unique identifier the client sends with the request. Before processing, the server checks: have I seen this key before? If yes, return the cached result. If no, process it and store the result alongside the key.

Idempotency keys should be client-generated (UUID or hash of the important fields), stored in a fast lookup store like Redis, and have a reasonable TTL β€” usually 24 hours to a few days, long enough to cover any retry window.

Nearly every critical mutation in a production system should be idempotent: payments, order creation, account changes, notification sends. If it costs money or has side effects, protect it with an idempotency key.

Tradeoffs

SafetyRetries become safe. Double-charges, duplicate orders, duplicate notifications disappear.
StorageYou must store keys and their results, typically in Redis with a TTL.
Key designPoorly designed keys can cause false deduplication β€” treating different requests as the same.
ComplexityServices must implement the check-then-process pattern atomically.

Engineering Edge Cases

β–Έ

The check-and-store must be atomic. A race condition between 'check' and 'store' can still cause duplicates.

β–Έ

Downstream calls from your service also need idempotency β€” idempotency doesn't automatically propagate.

What comes next

You've made individual requests safe. But your services are getting faster at producing work than your downstream systems can consume. Something needs to slow them down.