Design a payment system: ledgers, webhooks, and idempotency
Payment
Guided ModeStep 7 β Reconciliation: Catching What Slips Through
Request Flow
No system is perfect
Even with idempotency, webhooks, and an immutable ledger, edge cases slip through: gateway timeout after charge completes, webhook delivery failure (gateway had an outage), race conditions in async processing. The Reconciliation Service is the safety net that catches all of them.
Nightly batch comparison
The Reconciliation Service runs as a scheduled job at low-traffic hours. It fetches the gateway's transaction log for the past 24 hours and compares it line-by-line against the Ledger Service records. This is deliberately off the critical request path.
Three categories of discrepancy
(1) In gateway, missing from ledger β missed webhook, trigger manual review. (2) In ledger, missing from gateway β phantom charge, refund immediately. (3) Amount mismatch β currency or rounding bug, alert engineering. The Reconciliation Service classifies and routes each.
Why We Need This
Stripe's own documentation recommends reconciliation as the final defence. Even with at-least-once delivery and idempotency keys, distributed systems have known unknowns. Reconciliation is the only way to achieve financial correctness across system boundaries.
Key Insight
Reconciliation is not an engineering concern β it's an accounting concern. Run it on every business day, alert on any discrepancy, and resolve within 24 hours. The longer a discrepancy sits, the harder it is to trace.
Step 7 of 7
