Dual Writes

Pradeep P
2 days ago
3 min read

Series: Modern System Design · Layer 4 — Data

Layer 4 · Post 14 of 17

← Previous: Transactional Outbox → Next: CRDTs

Layer 4 — Data · Post 108 of 119

Two separate writes cannot both succeed or both fail together. One will commit and the other will not.

What you'll learn

Why try/catch around DB + Kafka (or DB + DB) is not a transaction
The failure window, and why retries make it worse without idempotency
Outbox, CDC, and 2PC as the real options — and which to pick

The idea in one minute

A dual write is updating two systems that do not share a commit:

ok = db.insert(order)
ok = kafka.send(order)     // not the same transaction

Or DB + Elasticsearch, DB + Redis, DB + second DB, Stripe + your table.

Any crash, timeout, or process kill between the two leaves one updated. You now have a split brain of records, not of leaders: search says no order, billing says yes.

Two Generals (Post 99) is why you cannot close that window with "one more ack" across two APIs. An outbox (previous post) or CDC (Post 50) makes one system the commit, the other a derived stream. 2PC (Post 46) tries to couple two resource managers — rarely available across Kafka + Postgres in the way people wish.

Why it matters

This is the most common silent consistency bug in microservices. It looks fine in tests because the race is small. It shows up as "the email never sent" or "search is missing 0.01% of documents" — until it is 2% after a bad deploy.

Interviews: naming dual write and outbox is a senior tell. Saying "we'll put them in a try block" is the junior tell.

How it works

The window.

Write A succeeds, write B fails → retry B (need idempotency) or compensate A.
Write A succeeds, write B unknown (timeout) → retry B may duplicate.
Write B succeeds, write A fails → you published an event for a row that rolled back. Consumers create ghost state. This order is especially bad if you send to Kafka before the DB commit.

Rule of thumb: commit the source of truth first (DB), derive the rest. Never publish "OrderCreated" before the insert commits — unless you are willing to have consumers see and then unsee (hard).

Retries without keys. Dual write plus "just retry both" double-charges and double-inserts. Idempotency keys on both sides, or don't retry blindly.

Cache as dual write. UPDATE users + DEL cache. If delete fails, stale cache. Prefer TTL, CDC invalidation, or version in the value. Cache-aside still has a window; keep it short and acceptable.

Not dual write: two statements in one Postgres transaction. That is one commit.

A simple example

INSERT INTO orders then producer.send("orders", json). Kafka is slow; request times out; the app retries the HTTP handler; second insert (no unique key) → two orders, one or two messages. Or insert once, send fails, no message → warehouse never ships.

Outbox: one commit, poller sends, consumer keys on order_id. CDC: Debezium reads the insert from the WAL — the insert is the event.

Common mistakes

Kafka transactions as a fix for Postgres + Kafka. Kafka EOS does not include your SQL unless you use a connector story that is still one log (Debezium) or an outbox.

"We'll sync Elasticsearch in the request." Dual write with a search engine. Index from CDC.

Compensating ad hoc. if kafka_ok and not db_ok: kafka_delete — delete can fail too. You invented a bad saga. Use a real one or an outbox.

Calling the outbox dual write because there are two inserts. They share a commit. That is the point.

How this shows up in real systems

The bug in every codebase: save(); publish();
Debezium / Dynamo streams / Cosmos change feed: escape hatches.
Stripe webhooks + local DB: the webhook is the other write; upsert on event.id.
Graph DBs + OLTP: same pattern; pick a leader store.

Recap

Two commits ⇒ a window. One will eventually disagree.
One source of truth + derived events (outbox/CDC), or a real distributed transaction where it exists.
Next: merging concurrent writes on purpose — CRDTs.