Amazon has recently announced S3 Object Lambda, a new serverless feature to add customized code and process data from S3 before returning it to an application. S3 Object Lambda works with S3 GET requests and uses AWS Lambda functions to modify data as it is being retrieved from the object storage.
S3 Object Lambda is designed to add transparency to the consumer of the S3 objects with the clients receiving a modified version without interacting with another endpoint and without changes to an existing product.
Source: https://aws.amazon.com/blogs/aws/introducing-amazon-s3-object-lambda-use-your-code-to-process-data-as-it-is-being-retrieved-from-s3/
The object retrieved through the S3 Object Lambda Access Point does not have to exist in the underlying bucket: every GET request is intercepted by a Lambda invocation and the response can transform an existing object or generate a response. Danilo Poccia, chief evangelist EMEA at AWS, explains it with an example involving pictures and thumbnails:
When retrieving an object using S3 Object Lambda, there is no need for an object with the same name to exist in the S3 bucket. The Lambda function can use information in the name of the file or in the HTTP headers to generate a custom object. For example, if you ask to use an S3 Object Lambda Access Point for an image with name sunset_600x400.jpg, the Lambda function can look for an image named sunset.jpg and resize it to fit the maximum width and height as described in the file name.
Among other use cases, Amazon suggests transforming data formats (JSON to CSV), compression and decompression of data and redacting sensitive information from an object in S3. The code of a Lambda function that can be used with S3 Object Lambda Access Points to control access or to redact personally identifiable information was made available on GitHub.
Source: https://aws.amazon.com/s3/features/object-lambda
Jeremy Daly, GM of serverless cloud at Serverless Inc and author of the weekly serverless newsletter Off-by-none, is impressed and tweets:
What kind of #serverless magic is this? Amazing.
Eoin Shanaghy, CTO at fourTheorem, wrote an article about fetching and transforming data with the new feature, highlighting as well the current limitations: it supports only GetObject requests - no support for PutObject or ListObjects - and the AWS CLI commands aws s3 do not work with the S3 Object Lambda Access Point. He concludes:
As with many new features, just because they’re there doesn’t mean you have to use them! We may not need the added complexity this kind of dynamic behaviour brings (...) The beauty of S3 is down to its simple-yet-powerful, key-object store design. At the same time, there are plenty of real use cases that can take advantage of the ability to generate and transform objects on the fly. S3 Object Lambdas are a really valuable addition to development on AWS.
Many comments on Reddit have been positive, suggesting as well use cases in cybersecurity. A user writes:
I can see this being used in cyber threat intelligence. Store an event, whether it’s an AWS event or something else, like CloudTrail logs and enrich the objects with fraud scores, GeoIP etc as you download them. Or just download the event raw.
The new feature is already available in most AWS regions and is charged according to the AWS Lambda compute usage, the data S3 Object Lambda returns to the application and the S3 requests that are invoked by the Lambda function.