Message Queues

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 2 — Communication

Layer 2 · Post 4 of 10

← Previous: Synchronous vs Asynchronous Communication → Next: Kafka

Layer 2 — Communication · Post 19 of 88

A message queue holds work until a consumer is ready. It smooths spikes, decouples services, and makes retries a first-class idea.

What you'll learn

Producer, queue, consumer — the three roles
At-least-once delivery and why consumers must be idempotent
What a queue is good for versus a log (Kafka) or pub/sub

The idea in one minute

A message queue is a buffer of work.

A producer puts a message on the queue ("send this email", "resize this image").
The queue stores it until someone is ready.
A consumer pulls (or is pushed) the message, does the work, acks it.

If 10,000 orders land in one second and you have 20 email workers, the queue grows and drains. The API does not have to wait for SMTP.

Classic names: SQS, RabbitMQ, ActiveMQ. Redis lists are a small-scale cousin.

Why it matters

Queues are how you turn a spike into a slope. They are also how you survive a worker crash: the message is still there if you didn't ack.

They decouple deploy schedules. The API team can ship without the email team being up, as long as the queue is up.

They are not a database. If you need to query "all unsent emails for user 9," a queue is the wrong shape.

How it works

Competing consumers

Several workers read from one queue. Each message goes to one worker (approximately). That is how you scale processing: add consumers.

Delivery guarantees

Most queues are at-least-once: if a worker dies after doing the work but before ack, the message comes back. You might send the email twice.

At-most-once: lose messages rather than duplicate. Rarely what you want for orders.

Exactly-once: advertised more than it is truly free. You get close with idempotent consumers + dedupe keys.

Layer 3 (idempotency, retries) exists because queues retry.

Visibility timeout / ack

SQS hides a message for N seconds while you work. If you ack, it is gone. If you don't, it reappears. Set N longer than the job, shorter than "stuck forever."

Dead-letter queue (DLQ)

After N failures, the message goes to a DLQ for humans. Without a DLQ, poison messages block the line or retry forever.

Ordering

Many queues do not preserve global order. FIFO queues (SQS FIFO, some Rabbit setups) order per key at a throughput cost. Don't assume "first in, first processed" unless you paid for it.

A simple example

Thumbnail pipeline:

Upload API writes the file to object storage, enqueues { "imageId": "..." }, returns 201.
Workers pull, resize, write variants, ack.
If a worker OOMs, another worker gets the message after the timeout. The resize must be safe to run twice (same output keys).

Black Friday: the queue depth is your dashboard. Scale workers on depth, not on API CPU.

Common mistakes

Using the queue as the source of truth. When you need history, search, or "what happened last Tuesday," you wanted a log or a DB.

Huge payloads in the message. Put a pointer (S3 key, row id) in the message; keep the blob in storage.

One queue for every job type with no routing, or a thousand queues nobody owns. Group by failure domain and SLO.

No backoff on poison. A bad JSON will hammer the DLQ path or the dependency. Validate early.

How this shows up in real systems

SQS + Lambda / workers: the AWS default.
RabbitMQ: routing keys, exchanges, more knobs.
Celery, Sidekiq, Hangfire: app-level workers on Redis/Rabbit.

Next posts specialize: Kafka is a log you can replay; pub/sub is many consumers each getting a copy.

Recap

A queue buffers work between producers and competing consumers.
Expect at-least-once; make handlers idempotent; use a DLQ.
Queues smooth load. They do not replace a system of record.