Designing a Payment System

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 7 of 26

← Previous: Designing a Distributed Scheduler → Next: Designing a Metrics Pipeline

Layer 6 — Modern systems · Post 69 of 88

A payment system must be correct under retries. Money movement is the classic case for idempotency, ledgers, and careful failure handling.

What you'll learn

Why a ledger (append-only entries) beats "UPDATE balance" as the source of truth
How idempotency keys and PSP webhooks stop double charges when the network lies
What authorize vs capture, and a state machine, look like on the write path

The idea in one minute

You rarely "talk to Visa" from the checkout monolith. You create a PaymentIntent (or equivalent), talk to a payment service provider (Stripe, Adyen, Razorpay), and record every state change in a ledger. The network will timeout. The user will double-click. The PSP will retry the webhook. Correctness is idempotency plus an append-only record of money.

If you cannot explain what happens when the charge API returns 503 after the bank authorized, you have not designed payments.

Why it matters

This is the interview where "eventual consistency" makes people nervous — and should. You can be eventually consistent on "receipt email." You cannot be sloppy on "did we take ₹499 twice."

Auditors and chargebacks need a history, not a mutable balance column you cannot reconstruct.

How it works

Clients send POST /payments with amount, currency, customer, method, and an idempotency key (client-generated UUID for this checkout attempt). The API inserts a payment in requires_action / pending or returns the existing one if the key was seen.
Ledger. Every intent to move money is a journal entry: debit/credit accounts (platform, merchant, pending_processor). Balances are sums of entries, optionally cached. You never decrement a row as the only record.
PSP call. Authorize (hold) then capture, or one-shot charge, depending on the product. You store the PSP's payment_intent_id. Timeouts: read the PSP before retrying a new charge. Retrying blindly is how you double-charge.
Webhooks. The PSP posts payment_intent.succeeded. You verify the signature, upsert by PSP id, append ledger entries if not already applied, mark the payment succeeded. Webhooks are at-least-once — the handler is idempotent on event_id.
Failure. User closes the tab: you still have a pending intent; a job expires or cancels it. Partial refunds are new ledger entries, not edits. PSP outage: queue captures, do not pretend success to the user.
Stores. Payment state machine in OLTP; ledger table; optional outbox to notify orders. Do not let the order service mark "paid" without a payment id that is succeeded.

PCI: you usually tokenize cards (PSP.js / SDK) so your servers never see PAN. In an interview, say that unless they asked you to store cards.

A simple example

The user hits Pay. The app sends key chk_9a. You create pay_1 and call Stripe. Stripe authorizes, then your HTTP client times out. The UI retries with chk_9a. You return pay_1 still pending, retrieve Stripe's intent, see requires_capture, capture once, append ledger "captured 499 INR." A duplicate webhook arrives; event_evt_44 is already processed — no second credit.

If the retry had been a new key, you might have opened pay_2 and charged twice. That is why the checkout client must reuse the key for one user gesture.

Common mistakes

UPDATE balance without a journal. You cannot explain chargebacks or rebuild state.

Retrying charge on timeout without a retrieve. Classic double charge.

Trusting webhooks without signature check or without idempotency on event id.

Marking the order paid from a client callback ("Stripe.js said success") without a server-side confirmation.

Mixing currencies in one integer column without a currency code. Rounding and FX need explicit entries.

How this shows up in real systems

Stripe PaymentIntents, Braintree, Adyen: the PSP is the processor; you still own ledger and order coupling.
Shopify, Amazon, Grab: internal ledgers, many PSPs, payouts as a second state machine.
Double-entry bookkeeping in fintech: the same idea with more accounts.

Layer 2 retries are dangerous here until they are keyed. Observability is how you notice "pending" stuck — next, metrics.

Recap

Idempotency keys, retrieve-on-timeout, and webhook de-dupe keep money from doubling.
A ledger is the source of truth; balances are derived.
Payments are a state machine plus a PSP, not an UPDATE.

Once money moves, you need to see the system: metrics pipelines.

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 7 of 26

← Previous: Designing a Distributed Scheduler → Next: Designing a Metrics Pipeline