Poison Messages

Pradeep P
2 days ago
3 min read

Series: Modern System Design · Layer 2 — Communication

Layer 2 · Post 12 of 14

← Previous: Delivery Guarantees → Next: Head-of-Line Blocking

Layer 2 — Communication · Post 96 of 119

A payload that can never succeed will retry forever unless you isolate it (DLQ, skip, poison-pill handling).

What you'll learn

What makes a message "poison" versus "the dependency is down"
Redelivery limits, dead-letter queues, and why infinite retry is an outage
How this interacts with at-least-once delivery and head-of-line blocking

The idea in one minute

A poison message (poison pill) is a record the consumer cannot process successfully, no matter how often it retries: bad JSON, unknown schema version, a user id that violates a new constraint, a payload that always throws.

At-least-once delivery will keep handing it over. Each attempt may crash the worker, roll back, nack, and — in a queue that is FIFO or a single-threaded consumer — block everything behind it.

[ poison ][ good ][ good ][ good ]
     ^
     consumer dies / nacks / retries
     good messages wait  (head-of-line)

This is different from "Postgres was down for 30 seconds." That should retry. Poison is the message itself is the problem.

Why it matters

Queues exist to absorb spikes (Post 19). They also absorb permanent failures unless you bound retries. A one-line schema bug can stall a whole payment pipeline at 3 a.m.

Interviews: they want a dead-letter queue (DLQ) and an operator path, not "we retry with exponential backoff forever" (Post 28). Backoff is for transient errors.

How it works

Classify errors.

Transient: 503, timeout, lock timeout. Retry with backoff. Do not count these as poison immediately.
Permanent: 400-from-your-own-parser, ValidationError, NULL in a NOT NULL column after a deploy. Do not retry without changing the payload or the code.

Bound redeliveries. SQS maxReceiveCount, Kafka consumer plus your own retry topic, RabbitMQ dead-letter after N nacks. After N, move the message off the hot path.

DLQ (dead-letter queue). A separate queue/topic for failed payloads. Alert on DLQ depth. Tooling to replay after you ship a fix, or to drop after you confirm junk.

Poison-pill in Kafka. A bad record in a partition blocks that partition's consumer if you throw before committing the offset. Catch, log, skip or write to a DLQ topic, then commit. Skipping is a product decision (loss vs stall).

Do not DLQ on "I don't know." A bug that throws on every message will fill the DLQ with your entire backlog. Circuit-break the consumer; fix the code; replay.

Idempotency still applies. Replay from DLQ will duplicate side effects if you did not key them.

A simple example

Order worker expects {"orderId": "abc", "cents": 1999}. A client ships {"order_id": "abc", "cents": "19.99"}. Parser throws. SQS redelivers until the queue's retention. Visibility timeout fires, another worker dies, pager storms.

Fix: after 5 receives, SQS moves it to orders-dlq. Dashboard pages once. Engineer maps the field, ships a more tolerant parser, replays that one message. The live queue kept draining good orders.

If the consumer was single-threaded FIFO and you retried without a receive limit, no later order in that group would ship. That is poison plus head-of-line (next post).

Common mistakes

DLQ with no alarm. Silent graveyard. Customers wait forever.

Treating all exceptions as poison. Database failover looks like poison for 10 seconds. Use error types.

Infinite in-memory retry in the client while the broker thinks you are fine. You still hold the message; you just burn CPU.

Schema registry "I'll just skip unknowns" without a DLQ. You dropped money. Log and park.

How this shows up in real systems

SQS redrive policy → DLQ. The textbook setup.
Kafka: ErrorHandlingDeserializer, retry topics (Spring), or manual skip; Confluent / MSK patterns.
Pub/Sub dead-letter topics, EventBridge DLQ, Azure poison queues.
Saga / workflow engines: a failed step parked for humans — same idea with a UI.

Recap

Poison = this payload will not succeed; retrying it forever is an outage.
Bound retries, DLQ, alert, replay after fix.
Next: even a slow (not poison) first item can stall the line — head-of-line blocking.