Dead-Letter Queues

A dead-letter queue (DLQ) holds messages that could not be processed successfully, so that a poison message does not block a queue or disappear silently. It is a holding area for investigation and, once the underlying problem is fixed, for reprocessing.

Where DLQs apply

• Amazon SQS queues, through a redrive policy on the source queue
• Amazon SNS subscriptions, where the DLQ must be an SQS queue
• AWS Lambda asynchronous invocations and event source mappings
• Amazon EventBridge rule targets, where the DLQ must be a standard SQS queue

Queue type rules

• A DLQ for a FIFO queue must itself be a FIFO queue
• A DLQ for a standard queue must be a standard queue
• A DLQ attached to an SNS subscription or an EventBridge target is an SQS queue, and EventBridge accepts only standard queues
• The DLQ must be in the same Region and, unless a resource policy says otherwise, the same account as the source

How messages get there

• SQS moves a message to the DLQ once it has been received more times than the maxReceiveCount set in the source queue’s redrive policy
• The redrive allow policy on the DLQ controls which source queues may use it
• Redrive moves messages from the DLQ back to the source queue after the fault is fixed — useful for transient failures such as a dependency that was down

Monitoring

Alarm on ApproximateNumberOfMessagesVisible on the dead-letter queue itself. That is the metric AWS recommends for DLQs, and it is the one that works: messages moved automatically by a redrive policy are not counted by NumberOfMessagesSent, so an alarm built on that metric will never fire for the case it was meant to catch. Only messages sent to the DLQ by hand appear in NumberOfMessagesSent.

Useful metrics for a DLQ:

• ApproximateNumberOfMessagesVisible — how many failed messages are waiting
• ApproximateAgeOfOldestMessage — how long the oldest has been waiting, which is the better alarm for “nobody is looking at this”
• NumberOfMessagesReceived and NumberOfMessagesDeleted — activity while draining

On the source queue, ApproximateAgeOfOldestMessage climbing while NumberOfMessagesDeleted stays flat is the signature of a message that keeps failing.

Investigating

• Read the message body and attributes; EventBridge DLQ messages also carry the rule ARN, target ARN, an error code, the exhausted retry condition and the retry count
• Check the consumer’s logs around the timestamp of the first receive
• Verify IAM permissions and, for cross-account delivery, the resource policy on the target
• Check whether the failure is the message (malformed input) or the environment (a dependency, a timeout, a throttle) — the fix and the redrive decision differ

Configuration practice

• Set maxReceiveCount high enough to ride out transient faults and low enough that a poison message is set aside quickly; 3 to 5 is a common starting point
• Give the DLQ a longer retention period than the source queue, so a failure over a weekend is still there on Monday
• Use separate DLQs for meaningfully different failure modes rather than one queue for everything
• Test the redrive path before it is needed in anger

Example: AWS CLI

# Create a DLQ
aws sqs create-queue --queue-name MyDeadLetterQueue

# Set up the redrive policy on the source queue
aws sqs set-queue-attributes \
    --queue-url https://sqs.region.amazonaws.com/account-id/MySourceQueue \
    --attributes '{
        "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:region:account-id:MyDeadLetterQueue\",\"maxReceiveCount\":5}"
    }'

# Move messages from the DLQ back to the source queue
aws sqs start-message-move-task \
    --source-arn arn:aws:sqs:region:account-id:MyDeadLetterQueue \
    --destination-arn arn:aws:sqs:region:account-id:MySourceQueue