Amazon SQS: Decoupling Applications

Amazon SQS (Simple Queue Service) is a managed message queue used to decouple applications.

A producer sends messages to the queue, and a separate worker consumes and processes them. The worker can run on EC2, ECS/Fargate, Lambda, or another environment. SQS provides the queue; it does not provide the worker.

Instead of doing everything during the web request:

Django can put the background work into SQS:

Django can return the response quickly, while the worker processes the tasks asynchronously.

How does the worker get the task?

For a worker running on EC2 or ECS/Fargate, the worker application typically polls SQS using the AWS SDK:

ECS Worker
    │
    │ ReceiveMessage
    ▼
  SQS Queue
    │
    │ message
    ▼
ECS Worker
    │
    ├── Process task
    │
    └── DeleteMessage

Workers normally use long polling, which lets SQS wait for messages instead of making constant empty requests.

Multiple workers can poll the same queue:

              ┌── ECS Worker 1
              │
SQS Queue ────┼── ECS Worker 2
              │
              └── ECS Worker 3

SQS distributes available messages among the workers. A message received by one worker becomes temporarily invisible to other workers through the visibility timeout.

This means you can scale the number of workers based on the amount of work:

More messages
     ↓
More ECS workers
     ↓
More tasks processed concurrently

With Lambda, your Lambda function does not normally poll SQS itself. An SQS event source mapping managed by AWS polls the queue and invokes the Lambda function when messages are available.

If processing fails and the message isn't successfully deleted/acknowledged, SQS can make it available again after the visibility timeout, allowing another attempt.

You can also configure a Dead-Letter Queue (DLQ) for messages that repeatedly fail.

Why use SQS?

• Decoupling → services don't depend directly on each other
• Buffering → absorbs temporary traffic spikes
• Asynchronous processing → work can happen in the background
• Independent scaling → workers can scale separately from the web application

SQS vs RabbitMQ

Both can provide the messaging layer between producers and consumers:

Django → SQS      → Worker
Django → RabbitMQ → Worker

The key difference is routing and delivery model:

• SQS → primarily a simple, managed queue; consumers typically poll for messages
• RabbitMQ → a message broker with richer routing capabilities, where consumers maintain connections to the broker and messages are delivered to them

So if you mainly need a reliable queue with minimal infrastructure management, SQS is often a natural AWS choice.

If you need more sophisticated message routing and messaging patterns, RabbitMQ can be a better fit.

Who can be the worker?

SQS doesn't require a specific worker technology:

SQS
 │
 ├──→ Lambda
 ├──→ ECS / Fargate
 ├──→ EC2 worker
 └──→ Celery worker

Celery is optional. It is a task-processing framework that can use SQS as its broker.

Simple mental model

Producer → creates work
SQS → holds the work
Worker → gets and processes the work

Amazon RDS Multi-AZ is designed to improve database availability and resilience.

RDS maintains a standby database in a different Availability Zone and synchronously replicates changes from the primary. If the primary becomes unavailable, RDS can automatically fail over to the standby.

Simple mental model

Multi-AZ → "What if my database fails?"
Read Replica → "What if I have too many reads?"

With RDS Multi-AZ, you generally don't switch between primary and standby in the application. RDS handles the failover behind the endpoint, and the application reconnects to the same endpoint.

Amazon RDS Read Replicas are read-only copies of an RDS database that help handle read-heavy workloads.

The primary database handles writes, while changes are asynchronously replicated to one or more read replicas. Applications can send read queries to the replicas, reducing the load on the primary.

Simple mental model

Read Replica → "I need more read capacity."
Multi-AZ → "I need higher database availability."

If the requirement says read traffic is increasing or the database is overloaded by reads → think Read Replicas.

If the requirement is automatic failover when the primary database or AZ fails → think Multi-AZ.

One important detail: because replication is asynchronous, a Read Replica can temporarily have replication lag and be slightly behind the primary.

For RDS Read Replicas, the application generally decides where to send reads and writes. The application might have separate database connections:

write_db  → primary
read_db   → read replica

But Multi-AZ is different

With RDS Multi-AZ, you generally don't switch between primary and standby in the application.

RDS handles the failover behind the endpoint, and the application reconnects to the same endpoint.

Simple distinction:

Read Replica → application chooses where reads/writes go.
Multi-AZ → RDS handles primary/standby failover.

AWS Auto Scaling is a set of AWS capabilities that automatically adjusts compute capacity based on demand. When traffic increases, AWS can add capacity (scale out), and when traffic decreases, it can remove capacity (scale in). This helps applications maintain performance during traffic spikes while avoiding the cost of running unnecessary resources during quiet periods.

For EC2, the main component is an Auto Scaling Group (ASG). The ASG manages a collection of EC2 instances and lets you define a minimum, desired, and maximum number of instances. For example, you might configure Min = 2, Desired = 2, and Max = 6. When a scaling policy detects high demand, the ASG launches additional instances; when demand falls, it can terminate instances. The ASG can also replace unhealthy instances automatically.

When an Application Load Balancer (ALB) is used, the ALB distributes incoming requests across the EC2 instances through a Target Group. The important part is that the Auto Scaling Group can be associated with the Target Group, so newly launched instances are automatically registered with it. The ALB performs health checks, and once a new instance is healthy, it can start receiving traffic. When an instance is removed during scale-in, it is automatically deregistered so the ALB stops sending new traffic to it.

Example

Suppose the application normally runs with two instances:

Target Group
 ├── EC2 #1
 └── EC2 #2

Traffic increases → the scaling policy triggers → the ASG creates EC2 #3:

Target Group
 ├── EC2 #1
 ├── EC2 #2
 └── EC2 #3  ← automatically registered

The ALB health-checks EC2 #3. Once it is healthy, the ALB can send requests to it.

Simple mental model

• ALB → distributes incoming requests
• Target Group → contains the instances that can receive traffic
• Auto Scaling Group → manages how many EC2 instances exist
• Scaling Policy → decides when to add/remove capacity
• Scale Out → add instances
• Scale In → remove instances
• Health Checks → help ensure only healthy instances receive traffic

Auto Scaling manages how many instances exist.
ALB manages where requests go.
The Target Group connects the two.

IAM Roles are especially useful whenever you want an identity to access AWS resources without giving it long-term credentials. They are commonly used by AWS services and applications, for example, an ECS task can assume a role that allows it to read from S3, or an EC2 instance can assume a role that allows it to publish logs to CloudWatch. Roles are also useful for cross-account access, where a user or workload in one AWS account assumes a role in another account. The key idea is: use a role when access should be temporary or delegated, rather than creating and distributing permanent access keys.

Two IAM concepts that often appear alongside roles are service-linked roles and iam:PassRole. A service-linked role is a special IAM role created and managed by an AWS service for performing actions that the service needs on your behalf. For example, an AWS service may create a service-linked role so it can monitor resources or perform required operations without you manually creating the role. iam:PassRole, on the other hand, controls whether an IAM principal is allowed to give an IAM role to an AWS service. For example, when you create an ECS task and specify an IAM execution role, the person or system creating that task needs permission to pass that role to ECS. Importantly, PassRole doesn't allow you to assume or use the role yourself it controls the ability to delegate that role to an AWS service.

AWS Organizations

AWS Organizations lets you centrally manage multiple AWS accounts as a single organization. Instead of putting everything into one AWS account, a company might create separate accounts for production, development, security, logging, or individual teams. Organizations gives you a central management layer where you can group accounts into Organizational Units (OUs), apply policies across accounts, and manage things such as consolidated billing. This separation is valuable because an AWS account becomes a natural security and administrative boundary.

For example, you might have a structure like this:

The important idea is that Organizations manages the accounts, while IAM manages identities and permissions within those accounts. Organizations becomes particularly useful as your AWS environment grows, because you can establish security and governance rules centrally rather than configuring every account independently.

Service Control Policies (SCPs)

A Service Control Policy (SCP) is an Organizations policy that defines the maximum permissions available to accounts or organizational units. An SCP does not grant permissions by itself. Instead, it acts as a guardrail that limits what IAM users and roles inside an account can do. For example, you could apply an SCP to your Production OU that denies the ability to disable CloudTrail, meaning that even an administrator in a member account cannot perform that action.

A useful way to remember this is: IAM policies say what an identity can do; SCPs define the maximum boundary of what an account can do. An IAM policy might say “this role can delete S3 buckets,” while an SCP can say “accounts in this OU can never delete S3 buckets.” Both policies must allow an action for it to succeed.

AWS CloudTrail

AWS CloudTrail records activity and API calls made within your AWS environment. It answers questions such as who performed an action, what action they performed, when it happened, from where, and what AWS resource was involved. For example, if an S3 bucket was accidentally deleted, CloudTrail can help you determine which identity made the API call and when it happened. CloudTrail is therefore an important part of security, auditing, compliance, and troubleshooting.

CloudTrail can record management activity across AWS services, and you can configure trails to deliver events to destinations such as S3. For security-sensitive environments, it's common to make these logs difficult for ordinary administrators to modify or delete and to send them to a dedicated logging/security account.

Organizational Trails

An organizational trail is essentially a CloudTrail trail configured from the AWS Organizations management account so that it applies across the organization's member accounts. Instead of configuring an individual CloudTrail trail separately in every AWS account, you can establish centralized CloudTrail logging for the organization. This is particularly useful when you want consistent auditing across production, development, security, and other accounts.

This gives you a centralized view of activity across your AWS environment. A common pattern is to have CloudTrail logs from multiple accounts delivered to a dedicated logging/security account, helping separate the people who operate workloads from the people responsible for auditing and security.

AWS Control Tower

AWS Control Tower builds on top of AWS Organizations and provides a more opinionated way to set up and govern a multi-account AWS environment. Instead of manually configuring Organizations, accounts, policies, logging, and security guardrails yourself, Control Tower provides an automated framework for establishing a landing zone a standardized starting environment for your AWS accounts.

In one sentence: IAM controls access within accounts, Organizations manages multiple accounts, SCPs constrain what those accounts can do, CloudTrail records what happens, Organizational Trails centralize that auditing, and Control Tower helps establish and govern the entire multi-account AWS environment.

An AWS VPC (Virtual Private Cloud) is your logically isolated network within AWS. It defines the overall network environment in which your AWS resources, such as EC2 instances, ECS tasks, EKS nodes, and databases can communicate. When creating a VPC, you define an IP address range using CIDR notation, such as 10.0.0.0/16. You then divide this larger address space into smaller subnets. A VPC can span multiple Availability Zones (AZs) within an AWS Region, while each subnet belongs to one specific Availability Zone. This allows you to organize resources and design for availability and fault tolerance.

A key distinction is between public and private subnets. A subnet is considered public when its route table has a route to an Internet Gateway (IGW), allowing resources with appropriate public IP configuration to communicate directly with the internet. A private subnet does not have a direct route to an Internet Gateway; resources in it can still access the internet through a NAT Gateway when outbound internet access is required. Other important VPC concepts include route tables, which determine where network traffic goes; Security Groups, which act as stateful firewalls for resources such as EC2 instances; and Network ACLs (NACLs), which provide stateless traffic filtering at the subnet level.

The simplest mental model is: VPC = your overall network, subnet = a smaller network segment inside the VPC, Availability Zone = the physical AWS location where a subnet resides. Public and private subnets are then used to control how resources communicate with the internet. For example, a common architecture places a load balancer in public subnets, application servers such as EC2/ECS/EKS workloads in private subnets, and databases such as RDS in private subnets. Route tables, Internet/NAT Gateways, Security Groups, and NACLs then control how traffic moves between these components and outside the VPC.

AWS IAM (Identity and Access Management) is the AWS service used to control who or what can access AWS resources and what actions they are allowed to perform. The main building blocks are users, groups, roles, policies, and permissions. An IAM User represents a specific person or long-term identity, while a Group is a collection of users that can share the same permissions. An IAM Role is an identity that can be assumed temporarily by users, applications, AWS services, or other AWS accounts. A Policy is a document that defines what actions are allowed or denied on which resources for example, allowing an application to read objects from a specific S3 bucket. In IAM, permissions ultimately come from policies attached to users, groups, or roles.

A useful way to think about IAM is “identity → policy → permission → resource.” For example, rather than giving an application an AWS access key belonging to a developer, you would typically create an IAM Role with a policy allowing the application to read from S3, and then allow the application to assume that role. AWS evaluates the applicable policies whenever an identity tries to access a resource. Explicit Deny overrides Allow, while access is generally denied by default if there is no applicable Allow. IAM therefore provides the foundation for the least-privilege principle: give each user, application, or AWS service only the permissions it actually needs.

ECS is generally more AWS-specific and has a simpler set of concepts, while EKS gives you the broader Kubernetes ecosystem and model.

The most useful mapping to remember is ECS Task Definition ≈ Pod specification, ECS Task ≈ Pod, ECS Service ≈ Deployment.

Just don't map ECS Service → Kubernetes Service despite the names, they serve different purposes.

EKS (Elastic Kubernetes Service) is AWS’s managed service for running Kubernetes clusters. Kubernetes is a platform for deploying, scaling, networking, and managing containers, while EKS provides the Kubernetes environment on AWS. The main building blocks you'll encounter in EKS are clusters, Pods, Deployments, and Services.

A Pod is the basic running unit in Kubernetes and can contain one or more containers. For example, a Pod could contain both a Node.js API container and a logging container. A Deployment defines the desired state of your application, such as keeping 3 identical Pods running, and handles replacing failed Pods and performing rolling updates. A Kubernetes Service has a different role: it provides a stable network endpoint for reaching the Pods and distributes traffic across them. EKS can run the Pods on EC2 instances or on AWS Fargate, where AWS manages the underlying servers. 

Compared with ECS, the concepts are similar but the terminology and responsibilities differ. An ECS Task Definition is roughly analogous to a Kubernetes Pod specification, an ECS Task is roughly analogous to a Pod, and an ECS Service is roughly analogous to a Kubernetes Deployment because both maintain a desired number of running application instances. However, Kubernetes separates the Deployment (managing Pods) from the Service (providing stable networking), whereas ECS uses the ECS Service as the primary mechanism for maintaining Tasks and commonly works with AWS networking components to expose them.