AWS Lambda now provides a way to control the maximum number of concurrent functions invoked by Amazon SQS as an event source. Setting the Maximum Concurrency, developers can determine the concurrency of the functions processing messages in individual SQS queues, simplifying the scalability of serverless applications.
An event source mapping (ESM) enables a Lambda function to subscribe to an SQS queue, scaling according to the number of messages in the queue. The elastic ESM polls the queue for messages and invokes a Lambda function accordingly, with the maximum number of messages to be processed simultaneously determined by the service limits for both services.
Although using SQS as an event source for Lambda is a common pattern in serverless architectures, determining the appropriate concurrency, visibility timeouts, and maxReceiveCount settings has been a challenge in the past. John Lee and Jeetendra Vaidya, solutions architects at AWS, explain:
When a large number of messages are in the SQS queue, Lambda scales out, adding additional functions to process the messages. The scale-out can consume the concurrency quota in the account. To prevent this from happening, you can set reserved concurrency for individual Lambda functions. This ensures that the specified Lambda function can always scale to that much concurrency, but it also cannot exceed this number.
Source: https://aws.amazon.com/pt/blogs/compute/introducing-maximum-concurrency-of-aws-lambda-functions-when-using-amazon-sqs-as-an-event-source
AWS has made available a repository containing a SAM application to test the Maximum Concurrency. The demo application deploys a Lambda function that runs for 10 seconds, an event source mapping with a maximum concurrency of 5, and an SQS queue with a visibility timeout of 20 seconds to simulate a high function concurrency with a small number of messages.
Yan Cui, cloud consultant and AWS Serverless Hero, comments:
This is awesome news! Finally a proper solution for the SQS over-polling problem that you can run into with Lambda. Previously the best workaround was to use FIFO queues and manage concurrency using group ids. Great to see an official solution for this!
Zac Charles, serverless engineer at Stedi, who called the issue "SQS overpull" in the past, now warns:
The new maximum concurrency feature is a real solution and is exactly what the AWS community has been wanting for years. Now that it’s here, you should definitely use it, but just be aware that you can still experience the SQS overpull issue if your available concurrency reaches zero.
The available concurrency of the function is determined by the difference between the concurrent executions quota, the total reserved concurrency, and the number of invocations currently in progress.
There are no additional charges to use the new feature. Developers can set the Maximum Concurrency for Amazon SQS using the console, CLI, SAM, or the SDK for Lambda, with a minimum supported value of 2 and a maximum of 1000.