SQS

  1. SQS fully managed service means, eliminates the need to set up, manage, and scale message queue systems.
  2. Amazon SQS allows to buffer data and process them asynchronously.
  3. It also has a direct built-in mechanism for retries and scales seamlessly.
  4. Pricing is based on the number of requests and the data transferred, ensuring cost-effectiveness.
  5. Reduces the operational overhead typically associated with on-premises message queue systems.

1. SQS Types

  1. Standard Queues:
    1. Best-Effort Ordering: Messages might not be delivered in the exact order they are sent.
    2. At-Least-Once Delivery: Ensures that every message is delivered at least once, though duplicates may occasionally occur.
    3. Throughput:
      1. Nearly unlimited messages per second, making it suitable for high-scale, distributed applications.
  2. FIFO Queues (First-In-First-Out):
    1. Exactly-Once Processing: Guarantees that each message is delivered and processed exactly once.
    2. Preserved Order: Messages are delivered in the exact sequence they were sent.
    3. Throughput:
      1. By default, supports up to 300 messages per second (300 send, receive, or delete operations per second) without batching.
      2. With batching, it supports up to 3,000 messages per second (300 batches of 10 messages each).
      3. Batch size is capped at 10 messages per batch for FIFO queues.
      4. High Throughput Mode: Can be enabled to support up to 70,000 messages per second without batching and even higher with batching.
      5. Example: If you need to process 1,000 messages per second and order is important, you would:
        1. Use 100 batches of 10 messages each (since the maximum batch size is 10).
        2. This would allow you to meet the 1,000 messages per second rate while maintaining message order.

2. Amazon SQS and Amazon Kinesis Together?

Amazon SQS and Amazon Kinesis can be used together in a system, and they complement each other. Kinesis can capture real-time data, while SQS can be used to buffer and queue messages before they are processed by worker systems at a later time.

For example:

  1. A Kinesis stream could collect data in real time from an IoT device.
  2. Once the data is processed by Kinesis, it could be sent to an SQS queue.
  3. A set of EC2 instances or Lambda functions could read from the SQS queue and process the data further.

3. Amazon SQS can handle real-time data

While Amazon SQS can handle real-time data for many workloads, it's important to consider latency, throughput limits, and scaling when using it in high-volume, real-time environments. For extremely high throughput or low-latency real-time applications, using Amazon Kinesis (for streaming data) or Kinesis Firehose may be a better fit.

4. Question: SQS Visibility Timeout

An image-processing company has a web application that users use to upload images. The application uploads the images into an Amazon S3 bucket. The company has set up S3 event notifications to publish the object creation events to an Amazon Simple Queue Service (Amazon SQS) standard queue. The SQS queue serves as the event source for an AWS Lambda function that processes the images and sends the results to users through email.

Users report that they are receiving multiple email messages for every uploaded image. A solutions architect determines that SQS messages are invoking the Lambda function more than once, resulting in multiple email messages.

What should the solutions architect do to resolve this issue with the LEAST operational overhead?

  1. Set up long polling in the SQS queue by increasing the ReceiveMessage wait time to 30 seconds.
  2. Change the SQS standard queue to an SQS FIFO queue. Use the message deduplication ID to discard duplicate messages.
  3. Increase the visibility timeout in the SQS queue to a value that is greater than the total of the function timeout and the batch window timeout. (Correct Ans)
  4. Modify the Lambda function to delete each message from the SQS queue immediately after the message is read before processing.

Explanation: Increasing the visibility timeout would give time to the lambda function to finish processing the message, which would make it disappear from the queue, and therefore only one email would be send to the user. If the visibility timeout ends while the lambda function is still processing the message, the message will be returned to the queue and there another lambda function would pick it up and process it again, which would result in the user receiving two or more emails about the same thing.