Retries, poison messages and dead letter queues

Why it matters

Some messages will never process: a malformed payload, a reference to a deleted record, a bug in the consumer for one edge case. Left in the queue, such a poison message is redelivered forever, consuming worker time and, on FIFO queues, blocking everything behind it. The dead letter queue is the standard answer, and the difference between teams that use it well and badly is whether anyone ever looks at it.

The idea

A dead letter queue (DLQ) is a secondary queue where the broker or the consumer moves messages that have failed too many times. The main queue keeps flowing; the failures are preserved for inspection and replay instead of being dropped or retried without end.

Retry well before giving up

1. 1

   Classify the failure

   Transient (timeout, 503, deadlock, rate limit) deserves a retry. Permanent (validation error, 400, missing required field, business rule violation) does not; retrying a permanent failure ten times just delays the DLQ. Consumers should distinguish the two and send permanent failures straight to the DLQ.
2. 2

   Exponential backoff with jitter

   Wait 1 s, 2 s, 4 s, 8 s and so on, capped, with random jitter so a thousand failing messages do not retry in lockstep and hammer a recovering dependency. Brokers implement this with delayed redelivery or per-attempt delay queues.
3. 3

   Bound the attempts

   A maximum receive count (commonly 3 to 10) after which the broker moves the message to the DLQ automatically. Choose it so total retry time covers a typical dependency outage but not much more.
4. 4

   Keep the attempt history

   Attach attempt count, last error and timestamps to the message or its metadata so whoever opens the DLQ sees why it is there without re-running it.

Ordering and the poison pill

On an unordered queue a poison message wastes retries and nothing else. On a FIFO queue or an ordered partition, it blocks everything behind it, because the next message cannot be processed until this one is acknowledged. The choices are to move it aside quickly (a low attempt limit on ordered streams), to park it and skip (accepting an out-of-order gap you must reconcile later), or to pause the partition and page someone. There is no free option; pick deliberately and document it.

Making the DLQ useful

A dead letter queue nobody reads is a slow data-loss mechanism. The operating rules:

• Alert on depth: any message in the DLQ is a defect or an incident. Page on the first one for critical queues; daily digest for low-stakes ones.
• Inspect without consuming: a tool to view payload, error and attempt history. Most managed queues offer this; build it if yours does not.
• Redrive: once the bug is fixed or the dependency is back, move messages back to the main queue in bulk. Consumers are idempotent, so replaying is safe.
• Discard deliberately: some messages are genuinely garbage. Deleting them should be an explicit action with an audit trail, not a TTL expiry nobody noticed.
• Retention: keep DLQ messages longer than the main queue (days to weeks) so a slow investigation does not lose the evidence.

Related patterns

A retry topic chain (retry-1m, retry-10m, retry-1h, then DLQ) implements backoff on log-based brokers that lack native delayed redelivery. A circuit breaker in the consumer stops processing entirely when a dependency is clearly down, instead of sending every message through its retry budget. A parking lot is a DLQ for the DLQ: messages a human has reviewed and deferred, kept separate from fresh failures.

Trade-offs

A DLQ costs a second queue, alerting, a viewer and a redrive tool, and buys a system where failures are preserved, visible and recoverable instead of lost or infinitely retried. The real cost is organisational: someone must own the DLQ and respond to its alerts. On ordered streams, every option has a downside and the trade is between blocking and gaps.

In an interview

Finish any queue design with the failure path: "Consumers retry transient errors with exponential backoff and jitter up to five attempts; permanent errors go straight to the DLQ. DLQ depth above zero pages the owning team; messages carry attempt history, and after a fix we redrive them, which is safe because consumers are idempotent." For ordered streams, add: "On the order-events partition we cap attempts low and park poison messages so one bad event cannot stall every order behind it."