In a recent blog post, Amazon announced that customers can now use the Amazon EC2 Inf1 instances on Amazon Elastic Container Service (ECS). The company promises the instances will be high performant and have low, predictable costs.
At the virtual re:Invent conference last year, Amazon launched the Inf1 instances, which are powered by AWS Inferentia, a custom chip built from the ground up by AWS to accelerate machine learning inference workloads. A few months later in June 2020, the company announced that customers can now use the Amazon EC2 Inf1 instances on Amazon Elastic Kubernetes Service (EKS) – and now on ECS.
The Inf1 instances are available in various sizes ranging from 1 to 16 AWS Inferentia Chips, with up to 100 Gbps network bandwidth and up to 19 Gbps Elastic Block Store (EBS) bandwidth. One Inferentia chip contains four NeuronCores – each implementing a high-performance systolic array matrix multiply engine, which massively speeds up typical deep learning operations such as convolution and transformers. Furthermore, the chip has a large on-chip cache, which reduces external memory accesses and saves I/O time in the process. And with several of these chips on an Inf1 instance, users can partition their model across them and completely store in the cache memory.
To run machine models on Inf1 instances, users need to compile them to a hardware-optimized representation using the AWS Neuron SDK – which they can install on their own instances. Furthermore, users can also leverage the newly available Amazon ECS Optimized Inferentia AMI, a new Amazon Linux 2 based AMI for the Amazon EC2 Inf1 Instances on ECS - documentation and tutorials are available for TensorFlow, PyTorch, and Apache MXNet in the AWS Neuron SDK repository.
Holger Mueller, principal analyst and vice president at Constellation Research Inc., told InfoQ:
Amazon is making good progress with its inf1 instances becoming better integrated with the rest of the AWS technology stack. Today it is Inf1's turn to work and be integrated with Amazon's Elastic Container Service (ECS), bringing together two key capabilities that fuel the creation of next-generation applications: AI and Containers.
Also, Mueller noted:
The combination of both is especially essential to support the deep learning ambitions of enterprises, which have to be container-based for scaling as needed and need to leverage the most efficient AI chips of the respective cloud they work on, in Amazon's case Inf1 - which was just announced late in 2019.
Currently, customers can deploy Amazon Elastic Compute Cloud (EC2) Inf1 instances on Amazon ECS in the US, Europe, and Asia Pacific AWS Regions with more to follow. Pricing details of the Inf1 instances are available on the pricing page.