Warehouse Throughput Simulator: Scaling Compute via SQS

Drawing on my experience managing high-volume logistics and yard routing, I built a cloud architecture that mimics physical warehouse throughput. In physical logistics, a surge in inbound trailers requires an immediate scale-up of associate labor. In the cloud, a surge of data requires auto-scaling compute resources based on queue depth.

The Architecture Flow

1. The Producer (Inbound Surge)

A Lambda function acts as the "Producer," simulating thousands of rapid package barcode scans. It drops these JSON payloads into an Amazon SQS standard queue. SQS acts as the physical receiving dock, holding the data securely until workers are available to process it.

2. Queue-Based Auto Scaling

This is the core architectural concept: Instead of scaling compute based on CPU utilization, the EC2 Auto Scaling Group is tied to a Custom CloudWatch Metric tracking the ApproximateNumberOfMessagesVisible in the SQS queue. If the queue length exceeds the processing capacity of the current instances, Auto Scaling provisions more EC2 instances to handle the load.

3. The Consumer (Processing)

The EC2 instances run a consumer script that pulls messages from the queue, processes the "package routing" logic, writes the final state to DynamoDB, and explicitly deletes the message from SQS to prevent duplicate processing.