AWS recently announced a recursion control feature in AWS Lambda that detects and stops Lambda functions running in a recursive or infinite loop. This feature is available for Lambda integrations with Amazon Simple Queue Service (Amazon SQS), Amazon Simple Notification Service (Amazon SNS), or when invoking functions directly through the Lambda invoke API.
Within Lambda, a detection mechanism is implemented to identify functions that seem caught in a recursive loop. When a function is triggered by the same event more than 16 times, Lambda will halt the subsequent invocation and route the event to either a Dead-Letter Queue (DLQ) or an on-failure destination. Furthermore, if configured, users will receive an AWS Health Dashboard notification with troubleshooting steps, email, or Amazon CloudWatch Alarms.
Diagram showing some examples of loops that Lambda can detect (Source: AWS Lambda Developer Guide)
The authors of an AWS Compute blog post on the feature explain that the feature only works when functions use one of the supported AWS SDK versions or higher:
Lambda uses an AWS X-Ray trace header primitive called "Lineage" to track the number of times a function has been invoked with an event. When your function code sends an event using a supported AWS SDK version, Lambda increments the counter in the lineage header. If your function is then invoked with the same triggering event more than 16 times, Lambda stops the next invocation for that event. You do not need to configure active X-Ray tracing for this feature to work.
In the same blog post, the authors provide an example, available on GitHub, where they explain that after the 16th invocation, Lambda will throw a RecursiveInvocationException to the ECMAScript (ES) modules (ESM), which stops invoking the Lambda function since the maxReceiveCount is exceeded. SQS moves the message to the source queues configured DLQ.
Suppose the recursive detection occurs and the Lambda function stops. In that case, the company advises fixing the underlying issue and setting the concurrency to 0 by choosing the "Throttle" button in the Lambda console or using the PutFunctionConcurrency API. In addition, users can disable or delete the event source mapping or trigger for the Lambda function.
AJ Stuyvenberg, a staff engineer at Datadog and AWS Serverless Hero, tweeted:
New: Lambda has recursive loop detection! Now your functions won't runaway to the maximum concurrency limit if you accidentally cause an infinite loop.
Marek Kuczyński, a solution architect at Wiz, responded with:
Wow - this is big!
Seen too many refund cases because people created an invoke loop.
Cost can run high earlier in Lambda functions when a recursive loop occurs due to misconfiguration or mistakes when writing lambda code. This is now prevented since the recursive loop detection is enabled by default on Lambda functions, and Michael Brewer, a VP architect fellow platform engineering at Fiserv, added in a tweet:
Now please add a hard spending cap limit. At least for developer accounts. Once the spending limit, the account should be frozen. Either that or sandbox accounts.
And finally, Jatin Mehrotra, a cloud engineer at Classmethod, stated at the end of a blog post:
Right now, this feature is not supported for other services like Dynamodb and S3 events. In the case of Dynamodb and S3, the only way to detect a recursive loop is by using Cloudwatch alarms. Please make a note of this.
Lastly, to turn off the feature, users must contact AWS Support. In addition, for using recursive invocation patterns, details are available in the Recursive patterns that cause run-away Lambda functions in the AWS Lambda Operator Guide.
