With machine learning, we can reduce maintenance efforts and improve the quality of products. It can be used in various stages of the software testing life-cycle, including bug management, which is an important part of the chain. We can analyze large amounts of data for classifying, triaging, and prioritizing bugs in a more efficient way by means of machine learning algorithms.
Mesut Durukal, a test automation engineer at Rapyuta Robotics, spoke at Aginext 2021 about using machine learning in testing.
Durukal uses machine learning to classify and cluster bugs. Bugs can be classified according to severity levels or responsible team or person. Severity assignment is called triage and important in terms of prioritization, where the assignment of bugs to the correct team or person prevents a waste of time. Clustering bugs helps to see whether they heap together on specific features.
Exploring the available data on bugs with machine learning algorithms gave him more insight into the health of their products and the effectiveness of the processes that were used.
According to Durukal, machine learning can also be used to automate code reviews and for the self-healing of broken test cases after the code has been updated.
InfoQ interviewed Durukal about how he applied machine learning in his daily work as a tester.
InfoQ: What are the challenges that testers are facing nowadays?
Mesut Durukal: Nowadays, we have smartphones in our pockets. It would have sounded crazy to think about this some time ago, yet we have already normalized it. The point is, we have lots of smart solutions in our daily life. We can control the temperature in our room by vocal commands over mobile phones. We can connect them to the navigation panel in our cars as well.
Now let’s check the reflection of this conjecture onto use cases. As applications and platforms are connected to various others, there are lots of integration interfaces. The same application can be installed on various platforms: Mobile, PC, and IoT. This leads to a wide scope to be verified on various platforms with numerous integrations.
Since we can use smart solutions anywhere, there are lots of use cases in many domains like automotive, industry, robotics, and healthcare. Hence, domain knowledge is required to test successfully; learning never ends.
Using smart solutions in our daily life this much, we are generating a huge amount of data. Data management is difficult. We as testers need to monitor activities to be able to fully trace progress.
InfoQ: How have you applied machine learning in testing?
Durukal: To summarize what I have tried:
- Triage of bugs: classify bugs according to severity levels
- Correct assignment of bugs: predict assignee of a bug according to past experience
- Future work: cluster bugs to see whether they heap together on specific features- prioritization of cases by relating to bugs
I was performing lots of monitoring activities. One of the most precious data sources were the bugs in the product. They were revealing clues about the health of the product and our processes. After careful observation, I was able to get insights into the reasons why we had escaped bugs, and understand from which functions we mostly had bugs. In this way, I was able to get an idea about the priorities and the likelihood of the cases finding a bug.
Bug management is crucial in terms of waste of time as well. For example, if you do the triage wrongly, then an urgent bug fix can be postponed since you had not prioritized it. I applied most of the algorithms including SVM, Naive Bayes, Random Forest, and many others to be able to compare their performances on the classification of bugs. Here is a snapshot of the results of all algorithms:
From the results table, we see that although most of them perform similarly, SVM and SGDC classifiers give the best performance. Moreover, clearly regression algorithms such as LinearRegression do not succeed since it is a classification problem.
Most of them finish processing in a very short time, except for AdaBoost.
After analyzing the results, I selected SVM as the running algorithm regarding performance and processing speed.
InfoQ: How can machine learning be used in software maintenance?
Durukal: Maintenance effort can be reduced by utilizing machine learning (ML) techniques in the sense that we spend a significant amount of time in code review sessions. If we teach best coding practices to machines, they can perform a code review in the name of us. As an example from deepcode.ai, if we somehow succeed in teaching that having magical numbers is an antipattern to the machine, then it can detect the magical numbers and warn us, "I learned this is an antipattern, so please fix it".
Particularly, people dealing with UI automation would agree how self-healing methods save a lot of time. Test cases can be broken after code changes; if they are detected automatically and resolution is automatically suggested, it is easy to apply a fix.
InfoQ: What benefits did you get from using machine learning?
Durukal: Personally, I have not applied solutions to fully replace human effort with ML; but instead, to improve the processes by collecting various data. Data mining is a crucial part of ML solutions. After precious data is collected, hidden patterns can be revealed. At this moment, the importance of ML is much more visible I believe, since we as humans have a limited capacity to analyze huge data.
For instance, in my study, I collected bugs created in a real project. In this study, I aimed for the proper triage of bugs; I noticed that people can prioritize bugs on different levels. I mean sometimes I was tagging a bug with P2, while for others it was P3. Therefore, especially in distributed teams, opinions can differ. To be able to build a standard, you have to look for all previously opened bugs. When I was creating a new bug, I was not checking all previously raised issues, because I cannot do that as a human being :) But that is something machines can do in seconds. In certain aspects, we can conclude that they may perform more consistently and ease our processes.
InfoQ: What will the future bring in machine learning for testing?
Durukal: Even today, we have complex computational environments. But if it gets more and more mature, we could integrate those applications in testing phases as well. For instance, analyzing requirement documents by OCR techniques could return valuable insights. Or associations to other platforms could be analyzed in a wider context.
Each single day, we have to perform more and in less time. Apparently, we will need to make use of ML more in the near future.
InfoQ: What have you learned?
Durukal: There is a famous discussion on the replacement of human tasks with robots, where sometimes machines or robots can beat humans. It can be an advantage or a disadvantage that robots don’t have emotions. The variety or diversity of people can be a weakness or a power, depending on the situation. At the end of the day, we would succeed if we somehow make the collaboration and work together.
