Risk-based testing improves the quality of the delivered stories and helps system testers to become part of the Scrum team, said Csaba Szökőcs, a product expert at Evosoft Hungary Kft. At TestCon Moscow 2019 he explained how they adapted classical risk-based testing to fit with their agile implementation by making it part of the sprint planning and definition of done.
Szökőcs presented how he supported teams to move testing activities from the end of iterations to the beginning. When they switched to Scrum a few years ago, their developer team didn’t really know how to provide user stories which were fully tested, and didn’t know what "potentially shippable" meant. They were used to releasing once or twice a year previously, along with a "system testing" phase (a.k.a. "bug fixing") that took several months. Now with Scrum, they tried to deliver stories without bugs, so they introduced more testing activities in the newly formed Scrum teams.
As the team had less testing experience, they tried to use the already-known methods: creating a small design at the beginning of the sprint, implementing it and when finished, testing it. It was a mini-waterfall model inside the sprint and had the same issues as the normal waterfall model, as Szökőcs explained:
If we found a bigger problem during testing, usually it was already too late to fix it correctly, and we had to live with the consequences: accepting the bug and fixing it later, reducing the scope of the story or introducing a quick solution with the cost of technical debt. Neither of these were good.
Szökőcs mentioned that he had learned about risk-based testing at an architecture workshop and had the idea that it could be useful on their Scrum teams. They tried it by collecting the risks of each user story before planning them into a sprint. That helped them to think about testing activities before the story was implemented, and to avoid many problems instead of reacting to them. They involved the system tester colleagues very early on in order to benefit from their experiences.
Risk-based testing dramatically changed how the stories were handled during the sprint, and improved the quality of the delivered stories, said Szökőcs. In addition, it was very motivational for the system tester colleagues as well, because they were able to influence how the stories were finally realized, he said. Many of these system testers are now part of the Scrum teams.
"We are using risk-based testing in some of the Scrum teams, but not everywhere; some teams have tried it and they have been using it for a long time already, some teams are not", said Szökőcs. He mentioned that they have adapted classical risk-based testing to their Scrum process so that it fits better with their agile implementation.
Szökőcs said that normally risk-based testing is done in testing teams to find the most valuable tests to execute when they only have a short testing period. In Scrum teams they are doing a separate and independent risk-based testing for almost each user story or epic; it is part of the definition of ready that the risks for the story are collected and prioritized. They are collecting the risks as part of the refinement, involving some experts of the specific topic (system testers, architects, product owners). Sometimes they also find some already-existing bugs in the system, and sometimes they need to change the users’ story as well, extending the acceptance criteria with some special scenarios, or splitting out scenarios that are very tricky, he said.
After collecting the risks, two colleagues prioritize them according to the probability and the damage of the risk by giving each one a number between 1 to 5. Multiplying these numbers will provide the exposure of the risks; all risks with a high exposure must be handled some way, said Szökőcs.
He mentioned that additionally, an experienced tester estimates the test effectiveness of each risk, answering how easy is it to handle this risk with a test? This is also specified in a range from 1 to 5, where 1 is impossible to test, and 5 is obvious. Multiplying these numbers by the exposure will result in the Test Priority Number (in a range of 1 to 125). A higher number indicates a risk that has a bigger impact and is easy to test, therefore it has the most value to start with. A low number indicates a low impact risk that is hard to test, thus it is better to ignore it as it has no real value, argued Szökőcs.
The prioritization helps the team to create a test strategy with valuable tests targeting the risks with the most impact, instead of testing only the happy day scenarios. They can use the priorities to decide which level they will test a specific risk on, said Szökőcs.
InfoQ spoke with Csaba Szökőcs after his talk at TestCon Moscow 2019.
InfoQ: What have you learned from the way that you prioritize risks at Evosoft?
Csaba Szökőcs: First, the prioritization is optional. If we skip it completely, simply collecting the risks is still very valuable. Some teams are working this way, and they are happy with that.
Second, do not overthink prioritization. We usually do it this way: try to find the risk with the lowest number and give it a 1, then the risk with the highest number and give it a 5. All other risks can be quickly prioritized in between in no time.
Third, the test priority really helps the teams to find the most valuable tests for the story, and this improves the overall quality of the delivered user stories.
InfoQ: How do you collect risks?
Szökőcs: We collect risks in the context of the user story by answering the question, "What can go wrong in this story?"
Also, we try to imagine that the new feature is already working and think about different scenarios; how it will be used or misused. We try to combine the new feature with other existing features – can it go wrong somehow?
Different non functional requirements, like performance, usability, and security can also be checked; are they affected, can they go wrong within this story? What can go wrong if the system is under high load? What if the system somehow becomes inconsistent?
To find valuable risks (not just impossible scenarios), we usually invite experts in that specific domain: a system tester, or an architect or a product owner from other teams. This is motivational for them as well, because they can also have an influence on the story before the team implements anything.
It’s important to mention that we are not collecting project risks, like what happens when someone on the team gets sick or we run out of budget. Also, we are not collecting test cases; it’s possible that we can cover several risks with only one test case, or we need several tests for just one risk. Test cases are derived from the most important risks later.
InfoQ: What are your tips for doing risk-based testing?
Szökőcs: Give it a try; start simple and improve it step by step.
At the beginning, it is enough if you collect some risks for each user story as part of the refinement process, and document them in a simple list. This will already help the team to feel the pain points of the story and focus on them.
Later, try to invite some experienced colleagues to that meeting, where you collect the risks.
If you are getting used to this, also try the prioritization. See if this helps the team in defining better or more valuable test cases.
Do not make the risk collecting meeting mandatory or too long – the team might be already overloaded with meetings; they do not need yet another one. Do it with the team members who are interested, and let others skip this altogether; they can check the results later.
And finally: keep it simple. The most value is in the discussion about the risks; do not forget that.
