According to Benjamin Bischoff, developers find new tools much more interesting than old ones, as they offer an opportunity to learn new technologies and approaches and to expand their tool belt. Using tools that have been around for decades, however, can save time and budget. When evaluating tools, it is more important to understand the problem to be solved than to jump straight into the tools.
Benjamin Bischoff will give a talk about the value of using timeless testing tools at QA Challenge Accepted. This conference will take place on September 28 in Sofia, Bulgaria.
In his talk, he will discuss biases that play a role when favouring a new tool:
For example, there is the "sunk cost fallacy". This means that you are tempted to use a new tool if you have invested a lot of time and money. This can be training time as well as budget spent on the tool itself or learning materials. It can therefore happen that you close your eyes to the disadvantages of a tool and use it against your better judgement.
According to Bischoff, familiarity is the main reason for using tools that have been around for decades. If the tools are already familiar, their use can save a lot of training time and budget, and also help to achieve goals more quickly. Also, there is a deeper understanding of the potential issues with these tools, which is not the case with new tools:
New tools may suddenly show limitations when they are run in production, which can lead to considerable additional work.
To find suitable tools, Bischoff suggests gathering thorough information and creating proofs of concept to find out which tool is ideal in which situation:
When we were looking for a tool for our API tests, we created and compared three different proofs of concept with different tools. My position at the time was that I was not a fan of the Karate framework. In the end, however, I was convinced by how quickly we achieved a good result with this framework. Looking back, this was simply a prejudice on my part because I simply didn’t have enough knowledge about this tool. That changed during the proof of concept phase.
According to Bischoff, it’s helpful to always bring a healthy dose of scepticism with you and not get carried away by the flashy and shiny hypes. In the end, it’s not about using certain things at the drop of a hat, but rather analysing more precisely what you need and how you can get the most out of technology.
InfoQ interviewed Benjamin Bischoff about testing challenges and using tools to address them and asked him about his experience with AI tools.
InfoQ: What main testing challenges have you faced?
Benjamin Bischoff: I have already had several different testing challenges, specifically technical challenges in the sense of "How can we map this requirement as tests?" and "What technical options are there for this type of test?" Of course, this also includes tools.
For example, we needed a tool that would simplify the testing of APIs for us. The Karate framework proved its worth for us here, as it has a simple syntax, but is very flexible thanks to its extension options. For UI-based end-to-end tests, Selenium fulfilled our criteria as a remote browser control. It was important that both desktop and mobile platforms could be operated and that features such as websites with multiple tabs and windows worked. We also wanted a solution that would accurately simulate users from the outside and not run in the browser like many other solutions.
On the other hand, there have also been challenges due to prejudices against certain roles, e.g. test automation engineers or QA engineers and their goals and intentions. But these problems have disappeared with time and experience.
InfoQ: What tools do you use to address these challenges, and how do you use them?
Bischoff: In our UI-based end-to-end test strategy, we mainly use our in-house Selenium test framework. For me, this technology still has many advantages over the hyped new tools, mainly that it uses the W3C webdriver standard so that all browsers are supported out-of-the-box. It does not need any custom browser versions or has to run within the Javascript context. That means that it behaves like a real user - and that’s what we want.
We also use Bash and Make in the CI/CD context, for example. We have a lot of experience with this and can achieve our goals quickly.
Of course, we also regularly look at new technologies and approaches and consider whether and how we can integrate them into our test strategy. For example, in terms of test frameworks, we have also evaluated Playwright and Cypress, but in the end decided against them due to the reasons mentioned before. However, if we find interesting tools in the future, we have no qualms about replacing existing solutions with them. However, these must offer considerable added value that outweighs the resources invested.
InfoQ: What have you experienced using AI tools?
Bischoff: I use AI tools on a daily basis, both professionally and privately. For development and automation, I often use Github Copilot, Google Gemini and ChatGPT. If you already have experience in a certain area, these tools can be really helpful. But you need to know exactly what you need and how to express it. More importantly, you need to know when a proposed solution doesn’t make sense or might be hallucinated, and express this in a follow-up prompt.
Gradually, you will get more and more useful ideas and solutions from AI tools the more you give feedback. I always say that you have to treat AI tools like advanced rubber ducks.