Amazon announced an update to their Simple Queue Service (SQS) – developers can now use SQS messages to trigger AWS Lambda Functions. Moreover, no longer are developers required to run a message polling service or create an SQS to SNS mapping.
Amazon SQS is a message queuing service and has been available in AWS for over ten years. It is fully managed by Amazon, and enables decoupling between services. It is network accessible, auto-scales, and offers pay-as-you-go operation. SQS provides two types of queues:
- Standard queues, which provide maximum throughput, best-effort ordering, and at-least-once delivery
- SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent
AWS Lambda is a compute service that enables developers to run code without provisioning or managing servers. The service has evolved over the past four years with Amazon adding more features and now the support for SQS. This means developers can now use the queuing services as a direct event source for Lambda to trigger functions. Currently, the Lambda triggers only work with standard queues.
A typical set up to use an SQS trigger in a Lambda function is to add it to a function and configure the execution role for the function with the appropriate permission for communicating with an SQS standard queue. Next, specify the name of the queue that will trigger the function and how many messages the function should accept at a time.
When the batch size is set to one, then up to ten Lambda executions will trigger simultaneously. However, when the batch size is set higher, only one Lambda execution will trigger, and several messages (based on the size) will be sent. Chris Moyes, VP of Technology - ACI Information Group, stated in a TechTarget article:
Additionally, developers can use this batch size in connection with Reserved Concurrency settings in Lambda to set how many messages can process at once, which can be helpful to throttle usage or prevent it from flooding other systems, such as content delivery to an FTP server.
AWS Lambda will delete messages from the SQS standard queue once the function consuming the messages is initialised. However, when a function fails or times-out, the message will reappear on the queue as specified by the visibility timeout set on the queue. Developers can configure the default queue visibility timeout, retries, and duration between retries.
AWS Lambda can scale horizontally when consuming messages, and does so automatically. According to the blog post on the announcement:
The automatic scaling behavior of Lambda is designed to keep polling costs low when a queue is empty while simultaneously letting us scale up to high throughput when the queue is being used heavily.
The Lambda triggers registered in SQS can be managed in the SQS console; users can also configure and edit the trigger here. The new feature is now available in all regions where Lambda is available, the documentation is up to date, and there are additional charges except for the Lambda service continuously long-polling the SQS queue – Amazon will charge the account for those API calls at the standard SQS pricing rates.