Recently Amazon announced that AWS Lambda customers can now enable functions to access Amazon Elastic File System (Amazon EFS). With the support for EFS, they can share data across function invocations, read large reference data files, and write function output to a persistent and shared data store.
Until recently, Lambda functions could only access 512 MB /tmp directory storage, which was sufficient for a lot of use cases. For machine learning use-cases, however, Lambda functions were not an option as models like TensorFlow, which are often Gigabytes (GBs) in size, cannot fit in the /tmp directory storage. Moreover, it wasn't an option either when Lambda functions processed large amounts of data (GBs) and needed to store it on the /tmp directory storage for easy access. Fortunately, now customers can mount a file system with the EFS support and provide a local path to read and write data at low latency.
To use AWS Lambda with Amazon EFS, developers will need an EFS Access Point, an application-specific entry point in an EFS file system. This entry point includes the operating system user and group to use when accessing the file system and file system permissions, and can limit access to a specific path in the file system. Application code and file system configuration are decoupled in this manner.
To leverage an EFS file system with Lambda functions, a developer needs to create a file system first using the EFS console, and specify an Amazon Virtual Private Cloud, which is necessary to allow the function to reach EFS mount targets. Next, the developer can add the access point and finally review the configuration before hitting create. Subsequently, the developer heads over to the Lambda console to specify the same VPC connection, its subnets, and security group before adding the file system in the new File system section of the function configuration.
Source: https://aws.amazon.com/blogs/aws/new-a-shared-file-system-for-your-lambda-functions/
When using EFS with Lambda functions, developers will have to deal with the performance of EFS. For throughput, each file system can be configured to use bursting or provisioned mode. Furthermore, there are multiple levels of security controls to consider, and as Danilo Poccia, chief evangelist (EMEA) at Amazon Web Services writes in his blog post about EFS support for Lambda:
The Lambda function execution environment and the EFS mount point uses industry-standard Transport Layer Security (TLS) 1.2 to encrypt data in transit. You can provision Amazon EFS to encrypt data at rest. Data encrypted at rest is transparently encrypted while being written, and transparently decrypted while being read, so you don’t have to modify your applications. Encryption keys are managed by the AWS Key Management Service (KMS), eliminating the need to build and maintain a secure key management infrastructure.
With the support of EFS for Lambda Functions, Amazon is further expanding access to EFS for its AWS Services. Earlier, the tech company only allowed EC2 instances to access EFS, yet later also for EKS for Kubernetes, and ECS and Fargate for containers.
Mikhail Shilkov, a software engineer at Pulumi and Microsoft Azure MVP, stated in a tweet:
AWS Lambda launched EFS integration, so you can now attach large persistent disks to functions and share them between different compute units! That opens up a ton of data-intensive scenarios!
Support for EFS in AWS Lambda is currently available in all regions where AWS Lambda and Amazon EFS are available, except for regions in China. Furthermore, there are no additional costs for using EFS with AWS Lambda – customers only pay the standard price for AWS Lambda and Amazon EFS.