Crowdsourced testing is a unique way of involving the crowd- meaning the real users/testers- into software testing under real world conditions. It helped Swisscom to find defects very early in the development process and increase the quality of products, argued Maja Schreiner. Gathering and summarizing all feedback can be a challenge; having developers involved helped to speed up test cycles and improve understanding of how the product is tested and how the testers are thinking.
Maja Schreiner, senior test manager at Swisscom, spoke about crowd testing magic at the Spring Online Testing Conference 2017. InfoQ is covering the conference with Q&As, summaries and articles.
InfoQ interviewed Maja Schreiner after her talk about crowdsourced testing.
InfoQ: What is crowdsourced testing?
Maja Schreiner: Crowdsourced testing (CT) is a unique way of involving the crowd, meaning the real users/testers, into software testing under real world conditions.
Not only do we know that nowadays your end users do not live in testing labs, but in the whole world - crowdsourced testing will certainly enrich your testing strategy if your projects meet some of these requirements:
- You are developing user-centric software where success is determined by user’s feedback
- You want to eliminate confirmation biases
- You need to achieve the test coverage for different
- mobile iOS and Android devices
- operating systems
- internet browsers
- You need to scale your testing resources in a very short time period (testers test concurrently)
- You have a fix budget and want to save some money: CT typically saves about 30% - 40 %
InfoQ: How did you introduce crowdsourced testing, and what where the challenges you had to face?
Schreiner: We’re developing our products by following Scrum, Kanban and DevOps principles. We have four development teams which deliver small increments on a weekly or two-weekly basis and our manual functional test cycles differ between daily cycles on our DEV environment during the sprint and 48 h regression test cycles on TEST environment towards the end of the sprint.
I’ve inherited the contract with an external CT company and team, since my predecessor actually introduced the CT into our project a few months before I started. Now we’ve been using that same CT company and setup for more than a year.
The challenges that I had to deal with were:
- Developers rejecting lots of found bugs, and saying that they’re not delivering lots of value
- Developers distrusting CT since some of the bugs we’ve expected to be found were missed – of course I myself was also not satisfied with that fact
- Communication between CT, Devs and DevOps team taking lots of time
- Lots of administrative and coordination time needed on my side and therefore not having enough time for strategic tasks
- Everybody started seeing me only as a CT coordinator and forgetting about other responsibilities I’m having being a test manager
Those challenges impacted everybody in the team and it was clear they need to be solved as soon as possible.
First, you need to decide where to start and usually it’s where it hurts you the most. For me it was in reducing the time spent for communication and coordination. I’ve put time-box blockers in my calendar which reminded me I should stick to my schedule at the same time allowing me to invest more time in improving the process.
That means mostly improving the input and all other needed information my crowdsourced testers might need in order to deliver even better output test results. For every test cycle you need to provide them with clear inputs:
- Test scope / products
- Test plan (regression)
- Known issues list
- Internal release notes
- Sprint review video demo & presentation
- Roadmap
I was tracking the improvements daily by e.g. following on number of high and critical defects found, on the time spent etc.
Every week you need to inspect and adapt your strategy. The products, features and test scope you’re testing with CT may vary every week and therefore the inputs you need to provide as well. Your testers may vary, too. You need to analyze the different results you’ll get and use them in order to improve the quality of your development process and the end product.
InfoQ: Which benefits did you get?
Schreiner: By improving the input to our test cycles, we significantly improved the output in form of test results. Our developers were also slowly willing to take part in the cycles themselves, by either helping me deliver the needed input or also by approving and checking the findings.
You can imagine that gathering and summarizing all feedback might be the biggest challenge and the most overwhelming cause depending on a product or particular increment it might happen over and over again. So, in this case having developers helping me with that speeded up the test cycles significantly and also improved developers understanding of how we test and how the testers are thinking.
We benefited a lot from having defects found very early in the development process and by that increasing the quality of our products.
InfoQ: What did you learn from doing it?
Schreiner: I’ve learned to never stop learning and improving.
Also, never forget how important exploratory tests are, since they:
- add lots of value and definitely more than regression:
- testers are more motivated and creative
- generally, more bugs found
- sometime regression is done "too fast" and some bugs get overlooked and missed
Still, don’t forget to test internally as well and don’t expect CT to solve all your testing, QA or software development problems. You will have to work on solving them independent of if you’re having internal or external testing resources.
And also, last but not least: never forget to star and reward the good testers
- Keep the best testers in every cycle
- Invite new testers in order to get the fresh point of view and feedback the "old" testers might miss after some time testing more or less the same product
InfoQ: What advice can you give if someone wants to use crowdsourced testing?
Schreiner: If your applications require extensive business know-how like in banking or insurance, crowdsourced testing might not be the best solution for your needs, but if you’re using it never forget to:
- Work on your overall test strategy and development process
- Always inspect and adapt your CT strategy and process
- Have patience with different testers and members of your development team. Encourage the discussion and knowledge exchange
- Never stop learning and exploring. Always learn on your mistakes and successes, go to meetups, read blogs and literature, exchange your knowledge with your peers and sync with your mentor and your bosses