The End-to-End Argument

Pradeep P
2 days ago
3 min read

Series: Modern System Design · Layer 7 — Patterns and problems

Layer 7 · Post 1 of 5

← Previous: SLOs, SLIs, and Error Budgets → Next: Conway's Law

Layer 7 — Patterns and problems · Post 115 of 119

The network or middleware cannot fully guarantee the application is correct. The ends still have to check.

What you'll learn

Saltzer, Reed, and Clark's argument, in production language
Why TCP checksums do not replace file hashes, and retries do not replace idempotency
When putting a check in the middle is still worth it (the "sometimes" clause)

The idea in one minute

The end-to-end argument (1984): if the application needs a guarantee (integrity, exactly-once effect, privacy), that guarantee can only be complete at the ends that understand the application's meaning. Lower layers can help (performance, a first line of defense) but they cannot finish the job — because the threat or the failure is outside their view.

App A  -- TCP --  proxies --  disks --  TCP --  App B
        "we checksum packets"
        A still hashes the file; B still verifies
        A still uses an idempotency key; B still upserts

TCP retries lost packets. It does not know whether your charge should run once. A service mesh retries HTTP. It does not know whether your handler is safe to retry. Encryption in the load balancer does not encrypt the app's secrets at rest.

Why it matters

This is the named reason Layer 2–3 tools are necessary but not sufficient. Two Generals: the channel will not give you common knowledge. Delivery guarantees: the broker will not make Stripe exactly-once. Dual writes: Kafka will not commit with Postgres.

Interviews: "the mesh handles retries" is a middle. The end is the handler + idempotency key. The argument tells you where to put the last check.

How it works

The strong form. Only the ends have the spec. "This PDF is the contract we signed" is not a TCP notion. Hash at generate and at open.

The "careful" form (the paper's own hedge). Middles can implement incomplete versions for speed: checksums drop most corruption so the app hash almost always succeeds; TLS stops casual sniffing so the app's extra crypto is cheaper. You still keep the end-to-end check if the requirement is strict.

Where people violate it.

Trusting a queue's "exactly-once" for a side effect (Post 95).
Trusting a reverse proxy's timeout as the business timeout (the app still runs).
Trusting a WAF as authorization (the API still checks the user).
Trusting RAID as backups (the ends: "can I restore the product?").

Where middles shine. Compression, batching, connection pooling, opportunistic retries with idempotent ends. The argument is not "dumb pipes only." It is "pipes cannot complete app correctness."

A simple example

File upload: user → API → S3.

Middle: HTTPS, S3 checksums, TCP.

End-to-end: client computes SHA-256 before upload; server (or a job) verifies after GET; user sees "verified" only then. Bit flips on a disk firmware bug that S3's internal checks missed (rare) still get caught. More commonly: a truncated proxy body gets caught.

Payments: mesh retries POST /charge. End: Idempotency-Key. Without it, the middle's helpful retry is a double charge. The argument predicted that.

Common mistakes

Reading it as "no retries in the proxy." Retries are fine if the end is idempotent. The paper allows helpful middles.

Putting the only checksum in the CDN. The origin still needs to know what it stored.

End-to-end encryption theater while the app logs PII in plaintext. The ends include your log pipeline.

Using the paper to reject all middleware. Meshes, CDNs, and checksums are the incomplete-but-useful layer. Keep them; add the app check.

How this shows up in real systems

TCP vs application hashes (the textbook example).
TLS termination at a LB: middle encryption; mTLS to the pod or app-level crypto if the threat includes the LB.
Kafka acks vs consumer idempotency.
Kubernetes probes: middle "is the process up"; end "can it serve this customer's query."

Recap

Correctness lives at the ends that know the spec; middles can only approximate.
Keep checksums, retries, TLS — and hashes, idempotency, authz in the app.
Next: the org-chart version of architecture — Conway's Law.