Amazon SQS: Decoupling Applications

Amazon SQS (Simple Queue Service) is a managed message queue used to decouple applications.

A producer sends messages to the queue, and a separate worker consumes and processes them. The worker can run on EC2, ECS/Fargate, Lambda, or another environment. SQS provides the queue; it does not provide the worker.

Instead of doing everything during the web request:

Django can put the background work into SQS:

Django can return the response quickly, while the worker processes the tasks asynchronously.

How does the worker get the task?

For a worker running on EC2 or ECS/Fargate, the worker application typically polls SQS using the AWS SDK:

ECS Worker
    │
    │ ReceiveMessage
    ▼
  SQS Queue
    │
    │ message
    ▼
ECS Worker
    │
    ├── Process task
    │
    └── DeleteMessage

Workers normally use long polling, which lets SQS wait for messages instead of making constant empty requests.

Multiple workers can poll the same queue:

              ┌── ECS Worker 1
              │
SQS Queue ────┼── ECS Worker 2
              │
              └── ECS Worker 3

SQS distributes available messages among the workers. A message received by one worker becomes temporarily invisible to other workers through the visibility timeout.

This means you can scale the number of workers based on the amount of work:

More messages
     ↓
More ECS workers
     ↓
More tasks processed concurrently

With Lambda, your Lambda function does not normally poll SQS itself. An SQS event source mapping managed by AWS polls the queue and invokes the Lambda function when messages are available.

If processing fails and the message isn't successfully deleted/acknowledged, SQS can make it available again after the visibility timeout, allowing another attempt.

You can also configure a Dead-Letter Queue (DLQ) for messages that repeatedly fail.

Why use SQS?

• Decoupling → services don't depend directly on each other
• Buffering → absorbs temporary traffic spikes
• Asynchronous processing → work can happen in the background
• Independent scaling → workers can scale separately from the web application

SQS vs RabbitMQ

Both can provide the messaging layer between producers and consumers:

Django → SQS      → Worker
Django → RabbitMQ → Worker

The key difference is routing and delivery model:

• SQS → primarily a simple, managed queue; consumers typically poll for messages
• RabbitMQ → a message broker with richer routing capabilities, where consumers maintain connections to the broker and messages are delivered to them

So if you mainly need a reliable queue with minimal infrastructure management, SQS is often a natural AWS choice.

If you need more sophisticated message routing and messaging patterns, RabbitMQ can be a better fit.

Who can be the worker?

SQS doesn't require a specific worker technology:

SQS
 │
 ├──→ Lambda
 ├──→ ECS / Fargate
 ├──→ EC2 worker
 └──→ Celery worker

Celery is optional. It is a task-processing framework that can use SQS as its broker.

Simple mental model

Producer → creates work
SQS → holds the work
Worker → gets and processes the work