Testing in the Cloud: Exploring the Practice

This article first appeared in IEEE Software magazine and is brought to you by InfoQ & IEEE Computer Society

Cloud computing remains a hot topic in the IT industry as a model that provides computing resources on demand with reduced administration costs.¹ Its main service layers are infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS). Cloud computing supports an everything-as-a-service (XaaS) delivery model, providing Internet services ranging from security and databases to storage and integration (for cloud-based testing resources, see this link).

Cloud computing is changing the way we deliver and use software; traditional software licensing is decreasing, whereas companies hiring software services is on the rise.² In 2010, Gartner Research predicted that 20 percent of businesses would have zero ownership of IT assets by 2012 and instead seek to acquire assets from the cloud³ (although no follow-up research that we’re aware of confirms these predictions). The growth of cloud-based services is evident, but clouds must still undergo development and testing before we can deploy them. As the development of cloud-based services increases, so will the need for testing their applications.

Testing in cloud computing has relatively low entry barriers.⁴ Cloud-based testing is on demand and billed per use. Many organizations already provide cloud-based testing services (see Table 1) such as performance testing, load testing, and Web-based application testing, as well as the testing of environments hosted in the cloud.
The primary benefit of cloud-based testing is reduced costs for putting up, maintaining, and licensing internal testing environments.⁵ For example, Amazon’s cloud tested a network management system for a voice over IP telephony system for less than US$130.⁶ Other benefits include the flexibility to acquire a testing environment as needed and global market access for both vendors and customers.⁵ Despite these benefits, other costs are less obvious and more difficult to evaluate: testing in the cloud requires special technical skills to generate test cases and scripts, and providing and monitoring security might also incur additional costs. In this article, we propose a practical roadmap for adopting cloud-based testing.

TABLE 1

Examples of cloud-based testing providers.

Research Process

To understand the dynamics associated with adopting cloud computing, we conducted interviews in 15 software organizations. We then selected eight for an in-depth analysis based on their relatively progressive use of cloud computing and testing (see Table 2)—the other organizations were either moving toward adopting cloud computing or just considering its use, so we chose not to analyze them for our study. We searched for any indications of advantages, opportunities, challenges, or problems within the relationship between cloud computing and testing.

TABLE 2

Organizations interviewed.

Effects of Cloud-Based Testing

We identified several aspects of testing that cloud computing affects. We classified them into three groups: the effects on actual testing, the effects on the delivery and support of testing services, and the challenges of testing in the cloud.

Actual Testing

Our interviewees’ comments helped us recognize several notable effects on testing.

More efficient performance testing. Cloud computing can carry out performance, scalability, and stress testing in a timely and economical manner. With the cloud, it’s easy to provision servers at a speed and cost that would be impossible with traditional methods. As the owner of Organization 1 said, “You can easily launch different kinds of open-source stacks and even licensed servers like Windows Server 2003 [or] 2008 [with] practically no time and money, and install testing software on these servers and run the tests.” Traditionally, a testing lab would be a large room with many servers running different variations of an operating system. This is usually a large investment that requires maintenance even when the servers aren’t in use. With cloud computing, we can acquire the necessary amount of servers as well as different variations of the operating system and testing environments; in other words, we can test faster. Then, we can decommission the servers when we’re not using them. Two interviewees discussed performance testing in the cloud:

You just cannot get 40 high-end, expensive servers for your use in the traditional manner. There’s no way to do that. So that’s clearly an area where using cloud computing enables you to clearly do some kind of testing that is not possible at all otherwise. For example, I did some performance testing where I had 40 Amazon instances running for generating something like 20,000 realistic user experiences. They were actually running [a] full embedded browser suite, so it was not a simple HTML-HTTP request testing, but [used] some fully flexed browsers with DOM and JavaScript support and test[ed] a rich Internet application. (Chief technology officer, Organization 2)

There’s a Web ticketing service selling tickets to events. And obviously that has . . . high-demand spikes. So they wanted to test their solution. Have they done everything properly? Have they used the platform to its fullest? And the way they did the testing was that they actually built themselves a separate testing platform in the Azure environment so that they were testing their cloud solution from [the] cloud. Because that way you could get enough load generated without paying for hundreds of test servers. (Architect, Organization 4)

Quicker testing. Overall, testing times have become shorter, which promotes faster development:

After developing a new feature, whatever it is, you can test-run it in, say, 10 different environments. And you have the results available immediately and you can use them in further development and so on. (Owner, Organization 1)

In a large project, if you are doing the testing on a cloud-based environment, that would shorten your testing cycle because you would be able to iterate more quickly between the development team and the testing team. (Architect, Organization 4)

End products would also be more tailored to business needs. Companies would spend less time acquiring the infrastructure for testing (and development), allowing more time to address the actual business:

I think time scales get shorter and then the development gets faster. On areas like customer relationship management [CRM], it’s almost like you would register to Hotmail that your company’s CRM is ready. And then you just work with us to understand what changes you need, but immediately the whole system is provisioned for you. (Founder, Organization 3)

More realistic test results. A cloud-based system can easily work across different operating environments, letting developers understand their services’ usability from the users’ perspective. The owner of Organization 1 found this beneficial:

I have these customers that I have given this service for use at a discounted price, and in return they give me feedback about the service, about possible bugs, or other deficiencies that they find.

When considering whether to test in the cloud, companies with existing internal test environments must make their decision based on what they find important. There’s a tradeoff between an internal test environment that’s probably smaller than the production environment would be and a cloud-based test environment that would be similar to the actual production environment. An internal test environment would be safe but would compromise the quality of the test results. On the other hand, a cloud-based testing environment generates more realistic test results but can raise data security concerns. Organizations must decide which is right for them:

Most companies say, “Well, we already have this environment. Let’s use it and be happy with the results it can give us.” But you see the advantages in using clouds as testing environments. And it will certainly be, at some point, probably [the] near future, that somebody will take that step. (Testing and methodologies director, Organization 7)

Testing Services Delivery and Support

The comments also gave us more insight on delivery and support.

Better availability of testing tools and options. Cloud computing lets organizations carry out technical feasibility studies while trying to identify the best test tools and environments:

You don’t have to wonder what you could do if you had a certain kind of environment. You can just get the environment, test it, verify it, and that’s done. It cuts a lot of speculation. (Chief technology officer, Organization 2)

On-demand testing also means that companies don’t have to purchase, say, a yearly license for testing tools. Instead, they can provision them from the cloud whenever needed, thus increasing flexibility. One tool that Organization 4 offers can debug cloud-based solutions. The architect explained that it works like an airplane’s black box: “You can . . . step back from the error spot and see what exactly went wrong with the actual code that you are developing.”

Readily available cloud-based tools and environments enhance agility in testing and, effectively, in development overall. Using agile development methods could result in a continuous, fast-evolving IT services market that addresses business needs faster and better tailors end products.

Improved developer–tester communication. When both development and testing are in the cloud, development and testing teams have equal access to the system. According to Organization 7’s testing and methodologies director, “If you give access rights to all teams, developers might not have the quickest access anymore to try a little bit of this and that to see if it works.” This can enhance interaction between teams. Development teams must promptly communicate changes in requirements so the testing teams have the correct parameters for tests.

Enhanced service delivery for vendors. Cloud-based development and testing follow an agile approach, which lets organizations address customer needs faster:

Because we do not need to set up huge environments and so on, we are able to kind of go and say, OK, these are the requirements, let’s build it, and after one or two weeks, let’s look at it. What do you like about it? What don’t you like? Make changes, and agree, OK, the product fits . . . the need. (Founder, Organization 3)

Cloud computing also lets vendors interact with each other. If a customer demands testing capacity beyond one vendor’s capabilities, the vendor can hire resources from other vendors. Organizations 1 and 3 had this type of interaction:

We have looked at Amazon Web Services also to kind of see how we can utilize that platform for our testing. Every now and then we feel that we would need a server or a couple of servers for our testing purposes. But then it doesn’t make sense for us to buy [from Amazon] when other Web services are so available and at low cost. You know [Organization 1], he has set up his site where you can also buy server time, so that’s an area we look at. (Founder, Organization 3)

From a business perspective, this facilitates a timely approach to meeting customer needs and is a win-win situation for the vendors and the customers.

Cloud-based Testing Challenges

Cloud-based testing requires testing of additional aspects and parameters. These might not necessarily be new, but testing in the cloud might exacer­bate them. For example, different cloud systems need a certain level of integra­tion and interoperability to work to­gether. One interviewee said that future cloud-based enterprise resource plan­ning systems will call for “integrations between different clouds with different technologies.” Two other interviewees emphasized load balancing, network la­tency, and multitenancy.

Security-related issues are a major concern, especially in test data manage­ment. Storing and handling test data should never compromise confidential­ity. One way to test cloud security is to selectively expose data to public clouds, as one interviewee suggested:

Use the cloud to host some typically less-secure stuff. So, not really any customer contracts, for example, but some test plan might take place in a Google group. (Testing and method­ologies director, Organization 7)

Test data management is critical due to variations in regulations across differ­ent geographical regions.

In our previous empirical study,5 we observed that organizations that develop safety-critical applications tend to value the possession of domain knowledge. For this reason, these orga­nizations are usually reluctant to con­sider cloud-based testing. Additionally, companies might be resistant due to budgets. Cloud-based testing provid­ers should provide transparent pricing models so that customers are equipped with sufficient information for budget­ing and cost estimation. Taking test­ing to the cloud can introduce signifi­cant changes to an organization. Thus, they need effective change-management strategies and procedures.

Cloud-Based Testing Roadmap

Organizations should evaluate whether cloud-based testing is a feasible invest­ment. We propose a simple, practical roadmap for establishing testing in the cloud.

Understand Cloud Computing

Cloud computing is an increasingly feasible choice for testing. Organiza­tions must develop an understand­ing of it within their unique contexts. They should evaluate risks, includ­ing those related to security and per­formance, so that they can prepare to mitigate them. Many cloud computing workshops, seminars, and forums are good sources of information and ave­nues for improving awareness, such as the Cloud Computing and Service En­gineering Workshop, the Cloud Computing Seminar, and the Cloud Computing World Forum.

Conduct Pilot Projects

Organizations can carry out pilot proj­ects to fully explore the potential ben­efits of cloud-based testing. Pilots are a less risky way to evaluate the viability of testing in the cloud. Organizations can experiment with response times and different load conditions for vari­ous tools they’re considering. To pilot testing in the cloud, you can test the cloud itself or host testing infrastruc­ture in the cloud and then test both cloud and noncloud applications.⁷ Pilot projects using these methods can help advance testing procedures.

Develop Elaborate Strategies

Another safe way for organizations to explore cloud-based testing is to come up with elaborate strategies. This can include establishing criteria for appli­cations for testing, potential cloud ven­dors, and test levels (for example, inte­gration and performance tests). Cloud testing vendors and quality assurance consulting firms can offer advice and direction for creating these strategies. For small- and medium-sized enter­prises (SMEs), cloud-based testing per­mits heavy test-related IT investments, even with limited financial resources. This means that SMEs can have com­petitive access to potential customer bases that usually belongs only to big companies. This can result in a more thriving market.

Enhance Team Interaction and Prepare for Complexities

Development and test environments in the cloud will continue to increase over time. In one of the organizations that was evaluating the cloud for its busi- ness, the interviewee mentioned that they anticipated all their development and testing to be cloud-based in the fu- ture. This will lead to a higher degree of interaction between development and testing teams, more thorough test- ing, and faster software development cycles. At the same time, organizations need to prepare for additional testing brought about by the complexities and requirements for cloud-based applica- tions and systems.

Enhance Research and Industry Cooperation

Many open concerns still exist, in- cluding application issues (which ap- plications are best suited for testing in the cloud); management issues (how to organize human resources for cloud- based testing), legal and fi nancial is- sues (how to manage test data across different global jurisdictions), and economic issues (how to devise appro- priate pricing models).⁸ Instead of ad- dressing these challenges in isolation, we urge researchers and practitioners to collaborate to establish cloud-based testing goals relevant to the software industry.

An increasing number of or- ganizations are accepting cloud computing as a model to deliver, test, and use software ow- ing to its growing number of services. We can continue to expect more cloud- based applications to form. With im- proved interaction between developer and tester teams, we can also expect the applications to improve in quality. The proposed roadmap could act as a starting point for organizations consid- ering adoption of cloud-based testing. Our future work aims at improving the roadmap by proposing example strate- gies for companies.

Acknowledgments

The ESPA project (www.soberit.hut.fi /espa), the Finnish Funding Agency for Technology and Innovation, and the companies men- tioned in the project website supported this study.

References

1. P. Mell and T. Grance, “The NIST Defi ni- tion of Cloud Computing,” US Nat’l Inst. Standards and Technology, Sept. 2011; http:// csrc.nist.gov/publications/nistpubs/800-145/ SP800-145.pdf.
2. Gartner, “Business Models in Global Software Ecosystem,” executive summary, Finnish Funding Agency for Technology and Innova- tion, August 2009.
3. Gartner, “Gartner Highlights Key Predictions for IT Organizations and Users in 2010 and Beyond,” press release, 13 Jan. 2010; www. gartner.com/it/page.jsp?id=1278413.
4. J. Foley, “The Cloud’s Next Big Thing: Software Testing,” InformationWeek, 23 June 2009; www.informationweek.com/ blog/229206200.
5. L.M. Riungu, O. Taipale, and K. Smolan- der, “Software Testing as an Online Service: Observations from Practice,” Proc. 3rd Int’l Software Testing, Verifi cation, and Validation Workshop (ICSTW 10), IEEE CS, 2010, pp. 418–423.
6. Z. Ganon and I.E. Zilberstein, “Cloud-Based Performance Testing of Network Management Systems,” Proc. IEEE 14th Int’l Workshop Computer Aided Modeling and Design of Communications Links and Networks (CA- MAD 09), IEEE CS, 2009, pp. 1–6.
7. European Commission, The New SME Defi ni- tion: User Guide and Model Declaration, Enterprise and Industry Publications, 2005; http://ec.europa.eu/enterprise/policies/sme/ fi les/sme_defi nition/sme_user_guide_en.pdf.
8. L.M. Riungu, O. Taipale, and K. Smolander, “Research Issues for Software Testing in the Cloud,” Proc. 2nd Int’l Conf. Cloud Comput- ing Technology and Science (CloudCom 10), IEEE CS, 2010, pp. 557–564.

About the Authors

LEAH RIUNGU-KALLIOSAARI is a researcher at Lappeenranta University of Technology. Her research interests include cloud comput- ing and its impact across different organizational contexts, including testing and quality assurance. Riungu-Kalliosaari has an MS in informa- tion technology from Lappeenranta University of Technology. Contact her at leah.riungu-kalliosaari@lut.fi .

OSSI TAIPALE leads a research project on software quality and test- ing at Lappeenranta University of Technology. His research interests include software testing and quality assurance. Taipale has a DSc in technology from Lappeenranta University of Technology. Contact him at ossi.taipale@lut.fi .

KARI SMOLANDER is a professor of software engineering in the Department of Information Technology at Lappeenranta University of Technology. His research interests include architectural aspects of systems development and organizational views of software develop- ment. Smolander has a PhD in computer science from Lappeenranta University of Technology. Contact him at kari.smolander@lut.fi .

This article first appeared in IEEE Software magazine. IEEE Software's mission is to build the community of leading and future software practitioners. The magazine delivers reliable, useful, leading-edge software development information to keep engineers and managers abreast of rapid technology change.