Tom Killalea, Vice President of technology with responsibility for infrastructure and distributed systems engineering at Amazon.com wrote an article on ACM queue on building scalable web services. He outlines guiding principles to building scalable web services with a lot of real-world examples, the core theme of which is “build only what you need”.
Caution: Early Optimization
Time and resources spent optimizing scalability might be better spent improving customer experience and driving traffic.
Embrace: Other People's Work
He explains that leaning on other peoples work for frameworks and infrastructure can reduce time to market and helps focus on providing customer value.
Three significant developments have contributed in different ways to this lowering of the barriers: the trend toward SOA (service-oriented architecture), the emergence of cloud computing infrastructure services, and the availability of Web application frameworks such as ASP.NET, Django, Rails, and Spring.
Caution: Overoptimization
He references Nicholas Nassim Taleb’s work on highly improbable and unpredictable events that have massive impact, and suggests using redundancy as a strategy for increased availability; the idea being that providing for redundancy as a means to balance load as opposed to just as a failover mechanism can be more cost effective than over-optimization for statistically less probable events.
Embrace: the Cloud
Tom gives an example of how Animoto, a social web application, hosted in the Amazon.com EC2 infrastructure could suddenly scale out on demand, all the way up to 3500 instances. Which in non-cloud infrastructure would incur considerable cost to maintain during lean traffic periods.
Caution: Target-driven Optimization
There are dangers in modeling expected traffic and then building a very precise scalability plan to meet that target. Good models are hard to build and can suffer from simplifying or happy assumptions that discount variability. […] if your Web service is successful, you'll eventually see greater demand than the target model suggests—maybe not this Black Monday or Super Bowl Sunday, but perhaps soon after and at a less anticipated hour.
Embrace: Ripping the Wings Off
“Beyond analyzing what broke first and why”, Tom says “We look for how the given application or service can make progress without the broken or missing pieces, and then rerun the test to determine the next breaking point”.
Tom concludes his article saying “The most difficult challenge in building a scalable Web service is how to handle the trade-offs between durability, reliability, performance, and cost effectiveness in the presence of failures and extreme concurrent access patterns”.
In addition to Toms article, the October 2008 issue has other great articles on building scalable services.