Recently AWS announced the general availability of faster standard retrievals from S3 Glacier Flexible Retrieval. According to the company, the retrieval can be up to 85% faster and applies to the Standard retrieval tier when using S3 Batch Operations.
Earlier, the company improved restore throughput by up to 10 times for Amazon S3 Glacier and S3 Glacier Deep Archive. In addition, by utilizing Amazon S3 Batch Operations, it's possible to automatically initiate requests faster, allowing organizations to restore billions of objects containing petabytes of data.
Channy Yun, a principal developer advocate for AWS, explains in an AWS news blog post:
Using S3 Batch Operations, you can restore archived data at scale by providing a manifest of objects to be retrieved and specifying a retrieval tier. With S3 Batch Operations, restores in the Standard retrieval tier now typically begin to return objects to you within minutes, down from 3–5 hours, so you can easily speed up your data restores from archive.
Result of a sample retrieval job with 250 objects, each sized 100 MB (Source: AWS News Blog)
Users can access the S3 Batch Operations through the AWS Management Console, AWS Command Line Interface (AWS CLI), SDKs, or REST API.
An example of a restore job with an S3InitiateRestoreObject job using the AWS CLI will look like this:
$aws s3control create-job \
--region us-east-1 \
--account-id 123456789012 \
--operation '{"S3InitiateRestoreObject": { "ExpirationInDays": 1, "GlacierJobTier":"STANDARD"} }' \
--report '{"Bucket":"arn:aws:s3:::reports-bucket ","Prefix":"batch-op-restore-job", "Format":" S3BatchOperations_CSV_20180820","Enabled":true,"ReportScope":"FailedTasksOnly"}' \
--manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820", "Fields":["Bucket","Key"]},"Location":{"ObjectArn":"arn:aws:s3:::inventory-bucket/inventory_for_restore.csv", "ETag":"<ETag>"}}' \
--role-arn arn:aws:iam::123456789012:role/s3batch-role
As shown above, S3 Batch Operations job activity is recorded as events in AWS CloudTrail. For tracking job events, users can create a custom rule in Amazon EventBridge and send these events to the target notification resource of their choice, such as Amazon Simple Notification Service (Amazon SNS).
Gy Panday, a senior product manager of Amazon S3 Glacier at AWS, mentions in an AWS Tutorials & Demos video another benefit of the S3 Glacier performance improvement:
Even your large workloads can now benefit from higher throughput available from restores in minutes.
Other public cloud providers like Microsoft and Google offer similar services like S3 Glacier. For instance, Azure Archive Blob Storage, yet that tier doesn’t provide fast retrievals capabilities. In comparison, the Google Cloud offers Storage Nearline storage class, a low-cost, highly durable storage service for data archiving, online backup, and disaster recovery that does provide fast retrieval.
Faster standard retrievals for Amazon S3 Glacier Flexible Retrieval are currently available in all AWS regions, including the AWS GovCloud (US) regions and China regions, at no additional cost - only a charge for S3 Batch Operations and data retrievals applies (details are on the S3 pricing page under the Requests & Data Retrievals section).
