During the latest AWS Storage Day event, Amazon announced the general availability of Mountpoint for Amazon S3. The new open-source file client provides through a file interface the elastic storage and throughput of Amazon S3, supporting data transfer at up to 100 Gb/second between each EC2 instance and the object storage.
Announced in preview earlier this year, Mountpoint for Amazon S3 is designed for workloads like data lake applications that perform sequential writes, sequential and random reads, and do not need full POSIX semantics. Jeff Barr, vice president and chief evangelist at AWS, explains:
Many AWS customers use the S3 APIs and the AWS SDKs to build applications that can list, access, and process the contents of an S3 bucket. However, many customers have existing applications, commands, tools, and workflows that know how to access files in UNIX style: reading directories, opening & reading existing files, and creating & writing new ones. These customers have asked us for an official, enterprise-ready client that supports performant access to S3 at scale.
Other common use cases for the new open-source client are machine learning training, image rendering, autonomous vehicle simulation, and ETL. Mountpoint supports data transfer at up to 100 Gb/second between each EC2 instance and the S3 bucket, translating local file system API calls to S3 object API calls and reducing costs compared to a managed network file system (NFS). Barr adds:
Under the covers, the Linux Virtual Filesystem (VFS) translates these operations into calls to Mountpoint, which in turn translates them into calls to S3: LIST, GET, PUT, and so forth. Mountpoint strives to make good use of network bandwidth, increasing throughput and allowing you to reduce your compute costs by getting more work done in less time.
In a popular Reddit thread, different users question how the new product compares to existing open-source s3fs and commercial ObjectiveFS products. Fernando Schubert, senior cloud engineer at Human Made, comments:
Interesting this is now formally supported, but s3fs/fuse was already doing this for a while, eager for a feature comparison and benchmarks.
Jukka Forsgren, senior solutions architect at AWS, highlights one of the goals of the AWS client:
At my previous company, I was on 24/7 on-call duty for a large number of customer AWS environments. One of the common alerts was related to stuck S3 mounts on EC2 Linux instances, which we often had to fix in the middle of the night. In light of this, it was great to notice that we just GA'd our own client called Mountpoint for Amazon S3, with stability as one of its primary design goals.
Mountpoint is not a general-purpose networked file system and comes with some restrictions on file operations. Built on the AWS Common Runtime (CRT) library, the rust-powered client can read files up to 5 TB in size, cannot modify existing files or delete directories, and does not support symbolic links or file locking.
Not all S3 storage classes are currently supported, with Mountpoint not working with the archive classes Glacier Flexible Retrieval, Glacier Deep Archive, Intelligent-Tiering Archive Access Tier, and Intelligent-Tiering Deep Archive Access Tier.
There are no specific charges for the use of Mountpoint; developers pay for the underlying S3 operations. Mountpoint is available in RPM format and the roadmap is available on GitHub.