Distributed systems: CAP, quorum, replication, and consensus
System Evolution
Idempotency
Retries create duplicates. Idempotency ensures that repeating an operation is safe.
Why the previous solution stopped being enough
Retries saved you from transient failures. But now your payment service is processing the same charge twice β once from the original request, once from the retry.
Architecture Diagram
Client
Payment Service
Idempotency Key
txn_a8f91bz
Dedup Store
seen before?
Process once
Return cached result
An idempotent operation is one where doing it multiple times has the same result as doing it once. DELETE a file β if you issue that command ten times, you still end up with no file. That's idempotent. Transfer $100 β if you issue that ten times, you've now sent $1000. That is not idempotent.
The classic horror story: a user clicks 'Pay'. The request goes through, the payment is processed, but the response gets lost in transit. The client times out and retries. The server processes the payment again. The user has been charged twice.
The solution is an idempotency key β a unique identifier the client sends with the request. Before processing, the server checks: have I seen this key before? If yes, return the cached result. If no, process it and store the result alongside the key.
Idempotency keys should be client-generated (UUID or hash of the important fields), stored in a fast lookup store like Redis, and have a reasonable TTL β usually 24 hours to a few days, long enough to cover any retry window.
Nearly every critical mutation in a production system should be idempotent: payments, order creation, account changes, notification sends. If it costs money or has side effects, protect it with an idempotency key.
Tradeoffs
Engineering Edge Cases
The check-and-store must be atomic. A race condition between 'check' and 'store' can still cause duplicates.
Downstream calls from your service also need idempotency β idempotency doesn't automatically propagate.
What comes next
You've made individual requests safe. But your services are getting faster at producing work than your downstream systems can consume. Something needs to slow them down.
