On the occasion of Selenium's 10th Birthday Celebration, a bunch of ThoughtWorkers have compiled an anthology of essays on testing approaches, tools and culture. This anthology of essays is available as an eBook, titled - "Perspectives on Agile Software Testing" which can be downloaded from ThoughtWorks website.
This is a short, but informative book which includes the evolution of testing tools, testing in agile environment, testing for mobile, BDD and continuous delivery for mobile applications. The book has several strong points like comparison of the different tools; it is short but informative, deep description of mobile testing including continuous integration and continuous deployment concepts etc.
Anand Bagmar starts with the explanation of selenium and its usage. It was 2004 when Jason Huggins, while testing an internal application at ThoughtWorks, created a JavaScriptTestRunner that changed the way automating the browser-based-testing is done. This then evolved into “Selenium” which was then made open-source.
Alabe Duarte and Fabio Maia explain testing for mobile. The mobile approach has challenges to test internet latency, 3G/4G connections, WiFi and geolocation features. In order to test web applications and keep unit tests running fast, they often use Test Double. It helps avoid that code communicates with any process or external dependencies that are not in our control. A very interesting example is Appium that allows anyone to write tests for mobile applications developed in different languages. This happens because Appium implements the Selenium webdriver API, where any other language can connect to it through a webdriver.
Prateek Baheti and Vishnu karthik explains BDD style of testing in mobile applications. In the BDD style of testing, test specification is expressed in a neutral language which is very domain-specific. BDD assumes tooling support, which can link the test specifications to the underlying implementation.
An example of a test scenario for messaging application mentioned by authors is:
- Login me as foo@foo.com
- I am sending the message “Lorem ipsum dolor sit amet” to user@user.com
- “user@user.com’ should receive a notification upon new message arrival
The above test doesn't deal with any lower level details, instead it describes the test scenario in natural language with clear domain terms. In this test case, receiving a notification is a domain term and the implementation of it would be different across platforms.
Gayathri Mohan describes continuous delivery for mobile applications. In this chapter, she explains CD in the context of testing - manual and automated for Mobile apps. She deeply describes CI and cross platform automation framework.
InfoQ spoke with the authors about the book and the best practices for testing.
InfoQ: What is new in this book for readers?
This book has a great collection of essays related to different dimensions of testing, including -
1. Anand Bagmar’s take on understanding if Selenium has come to age, how automation using Selenium relates to the Test Automation Pyramid, and provides tips on what today’s tester can potentially do to gear up for the future.
2. Daniel Amorim’s take on different dimensions of testing in an agile environment.
3. There is a big focus on mobile testing –
- Different types of testing to be done for Mobile devices (Alabe Durate and Fabio Maia),
- How to apply BDD style of testing in Mobile Applications (Prateek Baheti and Vishnu Karthik).
- Gayathri Mohan takes this to the extreme level of explaining how one could do continuous delivery for mobile applications.
- Vikrant Chauhan and Sushant Chaudhary talk about the various different Challenges in mobile testing.
4. Misconceptions of BDD – explained by Nicholas Pufal and Juraci Vieira.
5. Paul Hammant’s take on hiring Selenium QA people with different levels of expertise.
InfoQ: You mentioned test pyramid anti patterns in the book. Could you please explain them?
The goal of automation is to provide quick feedback to the team if functionality that was working before has broken because of a recent change.
Martin Fowler has explained about the test automation pyramid in his blog.
The Ideal test pyramid has maximum number of automated tests, closest to the source code i.e. at unit test level, and the least number of tests at the top of the pyramid i.e. functional UI (user interface) level. Also, the unit tests should test the granular piece of the product code, and the UI test should focus on the business functionality validation of the built / deployed product - say in the form of user journeys / user scenarios. However, achieving and maintaining a ‘good’ test pyramid required a lot of effort in terms of process and practices by the team.
A lot of teams do not invest as much as required in test automation (of all types). As a result, the easy route is taken but getting a separate team to just keep on writing tests at the UI layer, after the code has been developed. As a result, there is very little emphasis on automating tests at the lower layers in the pyramid. Also, as everyone is aware, UI automation is slow in implementing and execution, and is also very brittle. The tests may keep failing for any small change in the UI. As a result, many teams, though having invested so much in UI automation, spend most of their time in fixing / updating / maintaining the UI tests and a separate army of testers still need to keep repeating the regression tests, manually. Hence the Ice-Cream Cone Anti-Pattern is reached.
In some cases, the testing team is working in a separate silo of the organization. Or, the testing may be done by a different vendor / partner. They are not testing the product in a collaborative fashion as it is being built by the developers. Here the developers and testers do not communicate about what tests are automated, or not. As a result, there is a potential duplication of test cases by the developers and the testers. A bigger problem is that there may be quite a few test cases / scenarios that get missed in automation. This is the Dual Test Pyramid Anti-Pattern.
Fabio Pereira describes this Dual Test Pyramid Anti-Pattern in another interesting way and calls it the Cupcake Anti-Pattern. He has also explained some techniques on how teams can move from these anti-patterns to the ideal test pyramid.
InfoQ: It’s mentioned in the book that the role of a QA has become much more complex and requires a mindset shift as well as technical disciplines. Please put some light on this statement.
With teams following agile methodology, each person on the team needs to step up their way of working. The below diagram, taken from the Agile QA Process, shows various activities a QA needs to go through in each Iteration.
In a short iteration, this becomes very intense. Also, the QA typically lives 3 lives in this way of working -
- The past - to support what has happened in the last Iteration(s)
- The present - to ensure the “right” features are being built, and that they are being built “right”
- The future - to keep an eye out on what is coming up next in terms of development, and start thinking on how to test that.
Lastly, test automation is an essential ingredient in ensuring the success of the team and the QA plays a big role in the team achieving that. This requires not only good manual / exploratory testing skills, it also required a good technical understanding of the product-under-test, to ensure the correct tests are being identified and automated at the correct layer of the test pyramid.
With all these activities, and the multiple-lives the QA lives, he / she needs to optimize the way of working and be able to constantly re-prioritize appropriately on what is required to be done next. Thus, the role has got more complex, and warrants a different way of thinking in order to be successful.
InfoQ: You very well explained three dimensions of QA as business, technical and devops. What are all the similarities and differences in these three roles of QA?
In these three dimensions all of QAs will need to have ownership of product's quality and automation tests. All of them will need to keep the team focused on delivering a high quality product which is easy to maintain and implement new features.
There are some differences among these 3 sorts of QAs:
- In the business dimension, QAs will be driven by business and will have great communication skills to talk to clients; they will know a lot about specification by example and will have good active listening skill to extract acceptance tests from the clients.
- In the technical dimension, QAs have good programming skills and are experts on automation tests. They will be concerned about maintainability, always looking for practices like clean code, good practices and design pattern. These QAs pair with devs and build the solution together.
- In the DevOps dimension QAs will have knowledge about continuous delivery and repetitive tasks automation. They will help the team to build a test pyramid to allow a continuous delivery cycle.
InfoQ: The book talks about the Appium tool which implements the Selenium webdriver API, for mobile testing. How is this tool different from other mobile testing tools and what are the benefits of using it?
Appium has a great community. Because it is open source there is a lot of contribution from the community and that makes it to get new features and bug fixes faster. Also Saucelabs is supporting Appium if there is a need to run your tests in a cloud infrastructure for scaling purposes.
Because it implements a standard automation specification is easy to find a Selenium webdriver implementation for the language you are comfortable with so, for example, it makes simple for a QA team to use a language other than Java (Android) or Objective-C (iOS).
Other advantages are:
- If your mobile application is cross platform you are able to use the same set of tests for each platform.
- Appium is able to test the application that is closer to the production version, even better is that you are able to test the same version you intent to send to the marketplace with Appium.
InfoQ: How do you see the importance of continuous integration and continuous delivery in mobile application development?
In this fast-paced world, we, as business organizations, need to get our product and services in front of our users as quickly as possible. However, going quickly to market alone is not sufficient, if the product is not of good quality. Continuous Integration (CI) and Continuous Delivery (CD) are practices that give teams the confidence when taking the product live to customers.
These practices are applicable for Web Based products, as well as Mobile-based products. The difference lies in the tools, and the tweaks in the practices to get optimal results from the same.
InfoQ: What are the benefits of using BDD in mobile development?
The biggest benefit of BDD is the ability to talk the same domain / business language and as a result, remove any ambiguity in understanding amongst all stakeholders and team members.
If used correctly*, it hides the implementation details, and helps everyone focus on what is the business objective the feature / functionality is going to solve.
If implemented correctly**, it hides the platform specific details.
*: Like with any tool, you have to use BDD correctly, and in the proper context to get value out of the tool.
**: BDD is an additional layer of code (and tools) you are adding in your environment. With any such added tool / process, there is a cost factor associated with it. You should consider all pros and cons properly before deciding to use any tool / technology / process in the development cycle.
InfoQ: What should be the key skills of agile QAs?
Apart from having the requisite testing skills, the following skills (in no particular order) are very important for a QA to be successful in agile teams:
- Ability to prioritize various different testing activities.
- The mindset to help the team reach the ‘definition of done’ for each requirement picked up in the iteration.
- Negotiating with the business analysts or product owners about the true value of that requirement at the current stage - to ensure the “right” product is being built in the short time.
- Have an open mind - this will help learn new ways of working in a more optimized fashion.
- Help the team automate tests as low in the test pyramid as possible.
InfoQ: Do you have any plans to cover more agile testing concepts in your book in the near future?
There are a lot of ideas we have for the next book - to build on top of what this book covers. However, we are happy and willing to incorporate feedback and suggestions on what you would like to see more of in the next book!
About the Book Authors
Anand Bagmar is a hands-on and result-oriented software quality evangelist with 17+ years in the IT field. He has built open-source tools related to software testing – WAAT (Web Analytics Automation Testing Framework), TaaS (for automating the integration testing in disparate systems) and TTA (Test Trend Analyzer).
Daniel Amorim works as an agile consultant QA at ThoughtWorks. He's been working in the IT market since 2007 and is pursuing agile testing best practices in order to contribute with the development team.
Alabê Duarte is a consultant developer at ThoughtWorks since 2007.
Fábio Maia is a senior developer consultant at ThoughtWorks. He is particularly interested in desktop applications based in Qt/C++ or C#. Game developer based in C#/XNA
Prateek Baheti is an application developer at ThoughtWorks who has worked on Twist for over 2 years.
Vishnu Karthik is a developer at ThoughtWorks. He has been working on Twist and test automation.
Gayathri Mohan is a senior quality analyst at ThoughtWorks with 5+ years of experience. She has worked on multiple domains like Travel, Retail, e-Commerce and multiple platforms like .net, Rails, Android, iOS.
Vikrant Chauhan is a technology enthusiast. He is a keen follower of open source testing tools and tries to use them as and when possible.
Sushant Choudhary is a passionate technology lover and a sports enthusiast. Considering the ever changing technology scenario and its impact on the society, he is always keen on exploring and applying new developments in software engineering.
Nicholas Pufal has had a strong interest in programming since his adolescence, when he started dabbling in writing open source programs. He is passionate about design patterns, good development practices and ways to improve an agile team communication / delivered quality.
Juraci Vieira is a software enthusiast, who started his career as a tester. He was obsessed by automation, only to learn that quality is built in the software, by when he became a very passionate developer.
Paul Hammant is in his mid-40s. He’s worked for ThoughtWorks as a principal consultant since 2002, and has been consulting since 1989. As well as being nearly always billable to clients, he generally helps ThoughtWorks’ open source agenda.
