The Amazon Web Services (AWS) team announced a set of resources targeting the high performance computing needs of the scientific community. AWS specifically highlights their “spot pricing” market as a way to do cost-effective, massive scale computing in Amazon cloud environment.
Scientific organizations are frequently faced with the need to do compute-intensive activities when performing their analysis. In a case study highlighted by the AWS team, a top 5 pharmaceutical company was looking to perform molecular modeling against millions of compound targets. With help from Cycle Computing, this pharmaceutical company unleashed a 30,000 core cluster in AWS that completed their task in less than 8 hours. This cluster which was spanned two continents and had nearly 27 terabytes of RAM, cost $1279 per hour. Cycle Computing noted that their customer wouldn’t have even attempted to perform this scientific analysis in-house as it would have consumed all of their own data center resources for weeks. When an organization does attempt to accommodate this heavy-duty computing in-house, it often requires a large set of CPUs that sit idle waiting to be launched, as pointed out by Microsoft in this interview with Pharm Exec.
Look at all the data that goes into protein folding. Companies that are developing large molecule products–usually they are called monoclonal antibodies. The activity of that molecule is bound up in how it folds itself. During the discovery process they like to look at the primary sequence of that product and they want to do calculations about how it’s going to fold. They have traditionally maintained a large number of CPUs that have to be spun up around that activity and it can take 70 hours to finish up the whole protein folding analysis. With a cloud-based utility model, the servers don’t have to be on all the time, Something like that, that you don’t do all the time, scientists can work with the data when they need to, and the servers don’t have to be spinning when they are not needed.
While impressive in size and performance, this compute cluster managed by Cycle Computing was also cost-effective because of their use of Spot Instances, according to AWS. Unlike Reserved or On-Demand Elastic Compute Cloud (EC2) instances, Spot Instances are spun up as part of a bidding process. Customers define at which price per hour they are willing to pay for Spot Instances, and as long as the Spot price remains below the customer’s threshold, the Spot instances execute. As soon as the price threshold is crossed, the Spot instances are terminated. Spot pricing can be over 50% cheaper than Reserved or On-Demand instances which makes it a viable choice for either complimenting existing On-Demand workloads or defining a lower priority compute job that should only run when it is financially viable.
As part of their new “Spot and Science” page, the AWS team highlights the architectural considerations for dealing with transient compute resources such as those offered by the Spot market. AWS points out four architectural styles that accommodate solutions with potential for interruption including Map/Reduce, Grid, Queue-based and Checkpoint-based architectures. Each style either encourages small workload jobs that can be completed quickly and re-run if an executing host in terminated, or checkpoints that save work regularly.
The AWS “Spot and Science” page includes case studies, sample use cases, cost savings analysis, tutorials and architectural guidance. Even those outside the scientific community who wish to use the cloud for high performance computing can find relevant information in these use cases. From the HPC in the Cloud site:
The relatively high cloud adoption rate among pharmaceutical companies means that for other enterprises—those who have nothing to do with finding cures or improving health—can look to this industry for insights about real-world cloud use.
