Idempotency Design
Make retries safe so the system can self-heal without creating duplicates.
Why
Networks fail, clients retry, and a non-idempotent operation retried creates a duplicate — a double charge, a double send, a double order. Idempotency (same input → same result, no matter how many times called) is the property that makes retries safe, and retries are what make distributed systems reliable. Designing idempotency in from the start (idempotency keys on writes, dedup tables, deterministic IDs) costs hours; adding it after a double-charge incident costs weeks and a lawsuit. Every write that a client might retry needs to be idempotent.
When
For every write endpoint, before launch. Takes 1-2 hours per endpoint. Skip only for purely read endpoints.
How
1. For each write endpoint, ask: "if the client calls this twice with the same input, what happens?"
- Already idempotent (PUT with a fixed ID) → done
- Not idempotent (POST creates a new row each time) → add idempotency
2. Add an idempotency_key parameter (client-generated UUID)
3. On write, check: have we seen this key?
- Yes → return the stored result (don't re-execute)
- No → execute, store (key → result) with a 24h TTL
4. For internal events, use deterministic IDs (e.g. "{order_id}-charged") so redelivery dedupes
5. Test: send the same request 3x, assert exactly one side-effect
6. Document which endpoints are idempotent so clients know what's safe to retryExample
POST /v1/charges
body: { amount, customer_id, idempotency_key: "abc-123" }
First call: executes charge, stores (abc-123 → {charge_id: ch_1}), returns 201
Retry (network blip): sees abc-123, returns stored 201 with ch_1 (no second charge)
Third call: same — returns ch_1
Internal event: order.charged with id "{order_id}-charged"
Redelivery from Kafka → dedup table rejects → no double fulfillmentCommon Problems
- ⚠Assuming the client won't retry — they will, especially on mobile; design for it or accept the duplicates.
Best Practices
- ✓Make the idempotency key required on all non-safe POSTs — a 400 on missing key is better than a duplicate charge.
- ✓Store idempotency records in the same transaction as the write so a crash between them can't create a duplicate.