Delivery Guarantees

Pradeep P
2 days ago
3 min read

Series: Modern System Design · Layer 2 — Communication

Layer 2 · Post 11 of 14

← Previous: Hot Keys and Hot Partitions → Next: Poison Messages

Layer 2 — Communication · Post 95 of 119

What a broker can actually promise, and why exactly-once is really idempotent processing plus a write-ahead log.

What you'll learn

At-most-once, at-least-once, and what vendors mean by exactly-once
Why the network (Two Generals, later) makes "delivered once" a joint effort
How this should change your consumer code, not just a Kafka checkbox

The idea in one minute

When a producer sends a message, three different promises get marketed:

At-most-once. Send and forget, or drop on failure. Fast. You can lose messages. UDP, metrics fire-and-forget, some "best effort" logs.

At-least-once. Retry until the broker or consumer acks. You do not lose (if retries and storage work). You can duplicate. This is the default of SQS, RabbitMQ, Kafka without extra protocol, and HTTP retries.

Exactly-once. The effect happens once, even if the protocol duplicates. That is not a magic network property. It is dedupe + atomic side effects (idempotency keys, transactional outbox, Kafka transactions / EOS).

Producer --?--> Broker --?--> Consumer --> side effect (email, charge, row)
Each "?" can drop, delay, or happen twice.

Post 19 already said consumers must be idempotent. This post names the contract you are buying.

Why it matters

Interviews: "How do you not double-charge?" is this topic. "We use Kafka exactly-once" is a partial answer. Kafka EOS covers the log and the offset in a Kafka-to-Kafka (or Kafka-to-supported-sink) story. It does not make Stripe exactly-once. Your side effect still needs an idempotency key.

Product people hear "exactly once" and stop funding retries. Engineers hear it and still write INSERT without a unique key. Align the words.

How it works

Producer → broker. The producer retries. Without an idempotent producer (Kafka PID + sequence), a timeout after a successful write duplicates the record. With it, the broker drops the retry. That is exactly-once into the log, not into the world.

Broker → consumer. The consumer crashes after processing but before committing the offset (or deleting the SQS message). On restart the message is delivered again. At-least-once.

Closing the loop. To get an exactly-once effect:

Give the message a stable id.
Apply the side effect in a way that repeats safely (unique constraint, upsert, "charge id X").
Record "done" atomically with the effect (same DB transaction, or Kafka transaction for Kafka sinks).

If 2 and 3 are different systems without an outbox, you have dual writes (later). Then you are back to at-least-once and hope.

At-most-once is choosing to skip the retry. Use it when loss is cheaper than duplicates (some metrics, optional notifications).

A simple example

"Send order confirmation email."

At-most-once: worker dies after SMTP but you already acked — user never gets mail. Or you never retry a 503 — same.
At-least-once: you retry; user gets two emails unless email_send(order_id) is unique.
Exactly-once effect: table sent_emails(order_id PRIMARY KEY); insert then send, or send then insert with a saga. Duplicate deliveries no-op.

Kafka "exactly-once" between two topics: a consume-transform-produce in one transaction so the output record and the offset commit succeed together. The email SMTP call is still your problem.

Common mistakes

Believing the broker's slogan covers the database. Read the docs' scope.

Idempotency only in memory. Process restart forgets the dedupe set. Persist it.

At-most-once because duplicates were scary. Then silent loss. Prefer at-least-once plus a unique key.

Exactly-once across HTTP APIs you do not control. You ask for idempotency keys (Stripe, many payment APIs). That is the real protocol.

How this shows up in real systems

SQS: at-least-once (standard); FIFO has stronger dedupe in a window — still design idempotent consumers.
Kafka: acks, idempotent producer, transactions / EOS; consumers still replay.
Pub/Sub, NATS, RabbitMQ: at-least-once is the grown-up default; "exactly" is extra machinery.
gRPC / HTTP: retries + Idempotency-Key header — same three guarantees without a broker.

Recap

At-most-once can lose; at-least-once can duplicate; exactly-once is idempotent effects, not a perfect network.
Design the side effect for duplicates; then pick the broker feature that matches.
Next: when a duplicate (or a bad payload) never succeeds — poison messages.