A series of updates to AWS Lambda aim to improve how the function-as-a-service platform handles asynchronous workflows and processes data streams. These newly announced features arrived the week before the annual mega-conference, AWS re:Invent.
Synchronously calling a function means that Lambda executes the function and returns a response. Asynchronous invocations get sent to an internal queue and a separate process runs the function. If a developer wanted to send a message to a broker after completion of the async function, their choice was to use Step Functions, or write the code themselves within that function. With the new AWS Lambda Destinations, developers don't need to write any code to route results of an asynchronously-invoked function to an endpoint. Supported destinations include other Lambda functions, Amazon SQS, Amazon SNS, or Amazon EventBridge. The user can direct successful responses to one destination, and failure responses to another. The JSON-encoded result from the asynchronous function is sent as the "Message" to SNS and SQS, and as the payload to a Lambda function. AWS explained how this new functionality improves your event-driven architecture.
You no longer need to chain long-running Lambda functions together synchronously. Previously you needed to complete the entire workflow within the Lambda 15-minute function timeout, pay for idle time, and wait for a response. Destinations allows you to return a Success response to the calling function and then handle the remaining chaining functions asynchronously.
Image source: https://aws.amazon.com/blogs/compute/introducing-aws-lambda-destinations/
The team at Serverless says that Lambda Destinations also offer a more elegant solution for developers than what was previously available.
Considering the relative expensiveness of services like Step Functions, Event Destinations seems to be an excellent way to reduce both the complexity and cost of your serverless applications. It should allow you to nuanced workflows that were previously reserved for folks who were either willing to write that nuance into custom Lambda Functions, or who were willing to pay for and create Step Function workflows. That's not to say Step Functions has no place, it is still a great tool to visualize and manage complex workflows, but for more simple architectural needs Event Destinations seem like a great fit.
While some see features like Destinations as pure vendor lock-in, others celebrate the tighter integration between Lambda and other AWS services.
AWS also released three new capabilities related to data processing with AWS Lambda. First, Lambda now works with first-in-first-out (FIFO) queues in SQS. Lambda supported standard SQS queues since 2018, and now supports this queue type—first released in 2016—that retains message order. SQS FIFO queues rely on a pair of attributes sent in with the message: MessageGroupId which creates a collection of messages that get processed in order, and MessageDeduplicationId which uniquely identifies a message and allows SQS to supress messages with the same ID. According to AWS, "using more than one MessageGroupId enables Lambda to scale up and process more items in the queue using a greater concurrency limit." This model offers at-least-once delivery, and AWS says that if you need only-once delivery, you have to explicitly design for that.
Amazon SQS FIFO queues ensure that the order of processing follows the message order within a message group. However, it does not guarantee only once delivery when used as a Lambda trigger. If only once delivery is important in your serverless application, it’s recommended to make your function idempotent. You could achieve this by tracking a unique attribute of the message using a scalable, low-latency control database like Amazon DynamoDB.
The second data processing capability added to Lambda impacts how serverless functions scale to read events from Amazon Kinesis Data Streams and Amazon DynamoDB Streams. The Parallelization Factor can be dialed up or down on demand. AWS explained what this property does.
You can now use the new Parallelization Factor to specify the number of concurrent batches that Lambda polls from a single shard. This feature introduces more flexibility in scaling options for Lambda and Kinesis. The default factor of one exhibits normal behavior. A factor of two allows up to 200 concurrent invocations on 100 Kinesis data shards. The Parallelization Factor can be scaled up to 10.
Each parallelized shard contains messages with the same partition key. This means record processing order will still be maintained and each parallelized shard must complete before processing the next.
Image source: https://aws.amazon.com/blogs/compute/new-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources/
Relatedly, Lambda developers can now set a Batch Window property which specifies how many seconds to wait to collect records before invoking a function. AWS says this is useful when "data is sparse and batches of data take time to build up." It reduces the raw number of function invocations and makes each one more efficient.
The final data processing feature added to AWS Lambda gives developers more say in how to handle failures in batches of data. When Lambda reads data from Amazon Kinesis or Amazon DynamoDB Streams, it comes in sharded batches. Until now, if an error happens during processing of the batch, Lambda retries the whole batch until it succeeds, or the data expires. This means that no other data in the shard is processed while the offending batch undergoes retry attempts. Now, Lambda users have greater control of how errors and retries should get handled. By setting the MaximumRetryAttempts value, developers can dictate how many times to retry before skipping the batch. Relatedly, the MaximumRecordAgeInSeconds specifies how long to wait before skipping a batch. And BisectBatchOnFunctionError means the failed batch gets split and retries happen on smaller batches.
Image source: https://aws.amazon.com/blogs/compute/new-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources/