Amazon SQS: Decoupling Applications

Amazon SQS (Simple Queue Service) is a managed message queue used to decouple applications.

A producer sends messages to the queue, and a separate worker consumes and processes them. The worker can run on EC2, ECS/Fargate, Lambda, or another environment. SQS provides the queue; it does not provide the worker.

Instead of doing everything during the web request:

Django can put the background work into SQS:

Django can return the response quickly, while the worker processes the tasks asynchronously.

How does the worker get the task?

For a worker running on EC2 or ECS/Fargate, the worker application typically polls SQS using the AWS SDK:

ECS Worker
    │
    │ ReceiveMessage
    ▼
  SQS Queue
    │
    │ message
    ▼
ECS Worker
    │
    ├── Process task
    │
    └── DeleteMessage

Workers normally use long polling, which lets SQS wait for messages instead of making constant empty requests.

Multiple workers can poll the same queue:

              ┌── ECS Worker 1
              │
SQS Queue ────┼── ECS Worker 2
              │
              └── ECS Worker 3

SQS distributes available messages among the workers. A message received by one worker becomes temporarily invisible to other workers through the visibility timeout.

This means you can scale the number of workers based on the amount of work:

More messages
     ↓
More ECS workers
     ↓
More tasks processed concurrently

With Lambda, your Lambda function does not normally poll SQS itself. An SQS event source mapping managed by AWS polls the queue and invokes the Lambda function when messages are available.

If processing fails and the message isn't successfully deleted/acknowledged, SQS can make it available again after the visibility timeout, allowing another attempt.

You can also configure a Dead-Letter Queue (DLQ) for messages that repeatedly fail.

Why use SQS?

• Decoupling → services don't depend directly on each other
• Buffering → absorbs temporary traffic spikes
• Asynchronous processing → work can happen in the background
• Independent scaling → workers can scale separately from the web application

SQS vs RabbitMQ

Both can provide the messaging layer between producers and consumers:

Django → SQS      → Worker
Django → RabbitMQ → Worker

The key difference is routing and delivery model:

• SQS → primarily a simple, managed queue; consumers typically poll for messages
• RabbitMQ → a message broker with richer routing capabilities, where consumers maintain connections to the broker and messages are delivered to them

So if you mainly need a reliable queue with minimal infrastructure management, SQS is often a natural AWS choice.

If you need more sophisticated message routing and messaging patterns, RabbitMQ can be a better fit.

Who can be the worker?

SQS doesn't require a specific worker technology:

SQS
 │
 ├──→ Lambda
 ├──→ ECS / Fargate
 ├──→ EC2 worker
 └──→ Celery worker

Celery is optional. It is a task-processing framework that can use SQS as its broker.

Simple mental model

Producer → creates work
SQS → holds the work
Worker → gets and processes the work

Amazon RDS Multi-AZ is designed to improve database availability and resilience.

RDS maintains a standby database in a different Availability Zone and synchronously replicates changes from the primary. If the primary becomes unavailable, RDS can automatically fail over to the standby.

Simple mental model

Multi-AZ → "What if my database fails?"
Read Replica → "What if I have too many reads?"

With RDS Multi-AZ, you generally don't switch between primary and standby in the application. RDS handles the failover behind the endpoint, and the application reconnects to the same endpoint.

Amazon RDS Read Replicas are read-only copies of an RDS database that help handle read-heavy workloads.

The primary database handles writes, while changes are asynchronously replicated to one or more read replicas. Applications can send read queries to the replicas, reducing the load on the primary.

Simple mental model

Read Replica → "I need more read capacity."
Multi-AZ → "I need higher database availability."

If the requirement says read traffic is increasing or the database is overloaded by reads → think Read Replicas.

If the requirement is automatic failover when the primary database or AZ fails → think Multi-AZ.

One important detail: because replication is asynchronous, a Read Replica can temporarily have replication lag and be slightly behind the primary.

For RDS Read Replicas, the application generally decides where to send reads and writes. The application might have separate database connections:

write_db  → primary
read_db   → read replica

But Multi-AZ is different

With RDS Multi-AZ, you generally don't switch between primary and standby in the application.

RDS handles the failover behind the endpoint, and the application reconnects to the same endpoint.

Simple distinction:

Read Replica → application chooses where reads/writes go.
Multi-AZ → RDS handles primary/standby failover.