Distributed Transactions

Pradeep P
4 days ago
3 min read

Series: Modern System Design · Layer 4 — Data

Layer 4 · Post 6 of 11

← Previous: Eventual Consistency → Next: Two-Phase Commit

Layer 4 — Data · Post 45 of 88

A distributed transaction tries to make writes across multiple systems succeed or fail together. That coordination is expensive and fragile.

What you'll learn

Why two databases — or a database plus a queue — break single-node ACID
Dual writes as the original sin, and what "atomic across systems" actually costs
The two families of answers you will use in the rest of this layer: 2PC and sagas

The idea in one minute

A local transaction (posts 40–43) lives in one engine. A distributed transaction asks several systems to commit or abort as one:

[ Orders DB ] [ Payments DB ] [ Kafka ] \ | /

Networks fail. Processes pause. One participant says yes, another never answers. You no longer have a single WAL to replay. You have a coordination protocol.

If you skip the protocol and write twice (Postgres, then kafka.send), you have dual writes: one can succeed after the other failed. That is the bug this whole stretch of Layer 4 is about.

Why it matters

Microservices make this the default shape: Order Service owns orders, Billing owns invoices, Email is a Kafka topic. The user clicked once. The business invariant is still all-or-nothing — or at least "we can explain and repair the in-between."

Interviewers will let you say "transaction" once, then point at two boxes. Senior answers name the tradeoff: blocking atomic commit (2PC) versus local transactions plus compensation (sagas) versus accept eventual consistency and repair. Pretending XA will glue Stripe to DynamoDB is not a senior answer.

How it works

You have three honest options:

Don't. Keep the invariant in one database. An outbox table in the same Postgres transaction as the order row, then a publisher to Kafka, is "one local transaction + eventual copy" — not a distributed commit. CDC (end of this layer) is the grown-up version of that.
Atomic commit across participants (2PC / XA). A coordinator asks everyone to prepare, then commit. Strong, slow, and it can block if someone disappears after prepare. Next post.
Saga. Each service commits locally, then the next step runs. If a later step fails, you run compensating actions (refund, release inventory). Visible intermediate states. Eventual from the user's point of view. Post 47.

There is no fourth option called "try/catch around two HTTP calls" that restores ACID.

A simple example

Place order: decrement inventory in Postgres, charge in Stripe, enqueue a "OrderPlaced" event.

Stripe is not an XA participant. If you charge then crash before insert, you billed without an order. If you insert then crash before charge, you promised a shipment without money. A saga (reserve inventory → charge → confirm) with compensations (release, refund) is how most payment designs actually work. A single Postgres transaction can only cover your rows, including an outbox.

Common mistakes

Dual write and hope. db.save(); queue.publish(); with no outbox, no CDC, no idempotency.

Calling it a distributed transaction when you mean a distributed lock. A Redis lock around two DBs is not atomic commit. It reduces collisions; it does not give crash atomicity.

2PC across the internet and a third-party API. You do not 2PC with Stripe. You saga, webhook, and reconcile.

How this shows up in real systems

XA / JTA: Java EE and some app servers coordinating Postgres + JMS.
DynamoDB transactions: multiple items, still one AWS service.
Spanner / CockroachDB: look like one SQL database; they run distributed commit inside the product so you do not.
Stripe + your DB: webhooks, idempotency keys, reconciliation jobs — not 2PC.

Recap

Distributed transactions exist because one ACID engine is not the whole business action.
Dual writes are the failure mode. Outbox/CDC, 2PC, or sagas are the designs.
Next: two-phase commit — the classic atomic algorithm, and why it gets stuck.