Idempotency

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 3 — Reliability

Layer 3 · Post 4 of 14

← Previous: Exponential Backoff → Next: Circuit Breakers

Layer 3 — Reliability · Post 29 of 88

An idempotent operation can be applied more than once and still leave the system in the same state. That is what makes retries safe.

What you'll learn

What "same state" actually means for charges, rows, and messages
How idempotency keys work, versus naturally idempotent APIs (PUT by id)
Why at-least-once delivery (queues, webhooks, retries) forces this on you

The idea in one minute

Idempotent means doing it twice is the same as doing it once.

GET /user/9 is naturally idempotent. PUT /user/9 { "name": "Ada" } is too if it sets the name. POST /charges { "amount": 20 } is not: two POSTs are two charges unless you add a rule.

Client: charge $20, key=idem_abc try 1: timeout (server might have charged) try 2: same key --> server returns the first charge, does not charge again

Timeouts and retries (the last three posts) create duplicates. Queues and webhooks (Layer 2) deliver at least once. Idempotency is how the duplicate does not become a double spend.

Why it matters

Distributed systems do not give you exactly-once by default. They give you at-least-once plus "please make your handler safe."

If you cannot say what happens when the same request arrives twice, you do not have a reliable write path. Interviewers will replay the timeout-on-POST story until you mention a key, a unique constraint, or a naturally idempotent verb.

This is also a product contract: Stripe's Idempotency-Key exists because money is not allowed to be "best effort."

How it works

Three common patterns:

Natural idempotency. PUT a resource by client-chosen id. DELETE that is already gone returns 204. Upsert on a unique business key (order_id).
Idempotency key. Client sends a unique key (UUID) on a POST. Server stores key → response for a TTL (often 24 hours). Same key, same body: return the stored result. Same key, different body: 409. Persist the key in the same transaction as the side effect.
Dedup table / unique constraint. Duplicate idempotency_key selects the existing charge. Kafka consumers store an event id they already processed.

Writes need a dedup token both sides reuse on retry. A new UUID each attempt is how you double-charge with extra steps. Returning the original 201 on replay is correct; charging once and returning 500 is not (the client will retry again).

A simple example

Place order from a mobile app. The user has a bad network. The app sends POST /orders with Idempotency-Key: 7f3a....

First attempt: your API inserts the order, calls payments with their key, returns 201. The response never reaches the phone.
Second attempt: same key. You find the row, return the same 201. One order, one charge.

Without the key — or with a new key on each retry — support spends the afternoon on duplicate orders. Email workers: store event_id and skip if sent. SQS will redeliver.

Common mistakes

New idempotency key on every retry. The key must be stable for one user intent (one tap).

Storing the key after the side effect, without a transaction. Crash in between: you charged, forgot the key, retry charges again.

Idempotent HTTP and non-idempotent side effects. "We return 200 twice" while two Kafka publishes still fire. The whole apply must be covered.

Tiny TTL, or only the API is covered. Key expires before the webhook retries; or the worker double-sends. Cover every deliverer.

How this shows up in real systems

Stripe Idempotency-Key: the interview textbook. PayPal and Adyen have the same idea.
SQS / Pub/Sub / Kafka: at-least-once. Dedup in the consumer or use Kafka's idempotent producer for produce side (not a full exactly-once business story).
GitHub / Slack / Stripe webhooks: event ids. You already met this in Post 25.
Postgres unique constraints: the cheapest correct implementation for "this order id exists once."

Recap

Idempotent means retries do not change the outcome.
Use stable keys, unique constraints, or idempotent verbs — and persist them with the write.
Layer 2's queues and webhooks assume this. Without it, retries are unsafe.

Next: when retrying is the wrong move because the dependency is down, not blipping.

Series: Modern System Design · Layer 3 — Reliability

Layer 3 · Post 4 of 14

← Previous: Exponential Backoff → Next: Circuit Breakers