Amazon recently announced the general availability of File Cache, a managed high-speed cache for processing file data stored in disparate locations. The new service can be linked to multiple sources including on premises network file systems and managed AWS services like Amazon FSx or S3.
First unveiled at the AWS Storage Day 2022 last summer, Amazon File Cache is designed to provide low latency and high throughput for hybrid cloud workloads. The new service serves as a temporary, high-performance storage location for data stored in on-premises file servers or in cloud file systems and object stores. Sébastien Stormacq, principal developer advocate at AWS, explains:
Imagine you have a large data set on on-premises storage infrastructure, and your end-of-month reporting typically takes two to three days to run. You want to move that occasional workload to the cloud to run it on larger machines with more CPU and memory to reduce the processing time. But you’re not ready to move the data set to the cloud yet.
Source: https://aws.amazon.com/blogs/aws/amazon-file-cache-a-high-performance-cache-on-aws-for-your-on-premises-file-systems/
Among the bursting and hybrid use cases for File Cache, the cloud provider suggests media and entertainment, financial services, health and life sciences, microprocessor design, manufacturing, weather forecasting, and energy. Matt Gillard, technical principal at Contino, tweets:
Being able to mount multiple s3 buckets (and other sources) via NFS from on-prem or in the cloud opens many possibilities.
Stormacq adds:
Imagine another scenario where you have access to a large data set on Amazon Simple Storage Service (Amazon S3), spread across multiple Regions. Your application that wants to exploit this data set is coded for traditional (POSIX) file system access and uses command line tools like awk, sed, pipes, and so on. Your application requires file access with sub-millisecond latencies. You cannot update the source code to use the S3 API.
Customers are billed for the provisioned cache storage capacity and metadata storage capacity. In addition to File Cache costs, S3 requests, AWS Direct Connect charges, and data transfer fees apply. Andrew Guenther, principal software engineer at Orbital Sidekick, questions:
I saw the announcement and was hoping for a managed storage gateway. We use AWS Storage Gateway as an NFS interface for S3 with a 10TB cache and it costs ~$1k/month. That same cache is $13k on File Cache. What?! (...) Storage Gateway is way more flexible if you don't need the throughput of File Cache.
Corey Quinn adds in his newsletter:
I was worried when this was announced about what it would do to data transfer charges. If it automatically populates itself with files, couldn't it make decisions that were suboptimal, from a customer economics standpoint? I needn't have worried; AWS launched the service with a per-GB month price ($1.33!) that's so high that you won't notice or care about the usually-expensive AWS data transfer portion of the bill.
The announcement of the new service and the pricing model suggest that it is better suited for temporary high throughput workloads. Amazon File Cache is available in a subset of regions, including Ohio, Northern Virginia, Ireland and London.