SNS and Fan-out Architecture

Amazon SNS enables the publish/subscribe pattern: publishers send messages to a topic and SNS delivers them to every subscribed endpoint. It is worth reaching for whenever several processes must react to the same event independently and concurrently.

Core components

Topics are the communication channel. A publisher addresses the topic, never the subscribers.

Subscriptions attach endpoints to a topic. SNS delivers to:

• Amazon SQS queues
• AWS Lambda functions
• HTTP/HTTPS endpoints
• Email addresses
• SMS messages
• Mobile push notifications
• Amazon Data Firehose delivery streams

The fan-out pattern

The topic pattern and the fan-out pattern differ in what sits on the other side of the subscription. In the topic pattern, endpoints are notified directly. In the fan-out pattern, each subscriber is an SQS queue with its own consumer, so every branch buffers independently and can retry without affecting the others.

flowchart LR subgraph "Topic pattern" P1["Publisher"] --> T1["SNS topic<br/>Notifications"] T1 --> S1["Subscriber<br/>Mobile app"] T1 --> S2["Subscriber<br/>Email"] T1 --> S3["Subscriber<br/>Webhook"] end subgraph "Fan-out pattern" P2["Publisher"] --> T2["SNS topic<br/>Orders"] T2 --> Q1["SQS queue<br/>Order processing"] T2 --> Q2["SQS queue<br/>Inventory update"] T2 --> Q3["SQS queue<br/>Analytics"] Q1 --> L1["Lambda<br/>Process order"] Q2 --> L2["Lambda<br/>Update stock"] Q3 --> F1["Firehose<br/>Data lake"] end

Example: image upload processing

A user uploads an image, and three unrelated things must happen. The upload event is published once to an SNS topic, and each subscriber acts on its own:

1. Amazon SES sends the user a confirmation email.
2. An SQS queue holds the image for resizing by worker instances, which scale with the depth of the queue.
3. A Lambda function runs image analytics.

None of the three knows about the others. Adding a fourth — say, virus scanning — is a new subscription, not a change to the upload path.

Benefits

• Parallel processing of the same event by independent consumers
• Loose coupling: the publisher has no knowledge of who is listening
• Processing steps can be added or removed without changing the producer
• Each branch scales, retries and fails on its own

Practical notes

• Put a queue in front of anything that can be slow or unavailable. SNS pushes, and it will not hold a message for a subscriber that is down — it retries and then dead-letters or drops.
• Give each subscription a filter policy where subscribers care about only part of the traffic, rather than filtering after delivery.
• Make each consumer idempotent. Standard topics deliver at least once, so a consumer can see the same message twice.