Complex AI systems with non-deterministic outcomes pose challenges for testers and programmers. Such systems will increasingly become normal in high-impact, high-risk applications, argues Fiona Charles; testers should increase their capacity for thinking and learning and develop a number of personal strengths such as courage and good judgement.
Fiona Charles, a Software Test Consultant and Program Test Manager, will give the keynote presentation "Multiplying the Odds" at the European Testing Conference 2017:
Software daily increases its hold on the world, and we continually devise different ways to create and implement it, as well as embracing sophisticated tools for assisting every stage of the process. There is no longer a robust dominant paradigm of software testing. Our expectations of how to test, when to test, who should test — even whether we need to test at all — keep changing with new methods and technology.
InfoQ interviewed Fiona Charles about the main challenges that testing has to deal with and how these challenges impact the test profession, the value that testers are expected to deliver, essential skills for testers and how to acquire them, and what the future will bring for testers.
InfoQ: Which are the main challenges that testing nowadays has to deal with?
Fiona Charles: I don’t think the basic challenges are any different from those we’ve always had in software testing. We never have enough time or capacity to test everything, so how do we decide what to test, and how thoroughly to test so that we deliver optimal value to our stakeholders? Then having made those strategic decisions, how can we do that testing most effectively and efficiently?
The coverage challenge is universal. Outside the mainstream—as yet—there is also a significant challenge in the proliferation of complex AI systems with non-deterministic outcomes. These systems will increasingly become normal in high-impact, high-risk applications and I think more software practitioners will need to step up to both programming and testing them. As just one example, we’re seeing AI systems used now to make decisions about sentencing and parole in the US justice system. We’ve all learned from the media that "learning" systems are vulnerable to biases in the data selected for them to learn from. It should also be obvious that they are very challenging to test. And they will bring ethical challenges for programmers and testers alike.
InfoQ: How do these challenges impact the test profession?
Charles: The practice of professional testing is morphing, partly to meet the testing challenges, but mainly prompted by advances in technology and software development methodologies. I think it has always morphed and for similar reasons.
Certainly the advent of agile has changed where testers fit in a team and what they do on software projects. The old, so-called "waterfall" methods often misused testers’ time and skills as they busily sat about designing tests for weeks or months, only to find themselves mired in silly bugs because nobody or almost nobody had done any unit testing. Those horrid old days have gone for most testers (at least on the surface).
I’m not convinced that agile has really delivered on its promise of high-quality software, but the return to more integrated and collaborative teams of programmers and testers is mostly positive. I call it a "return" because that’s where many of us oldies started testing: on integrated, collaborative teams, where everybody pitched in and did what they could to get software we could be proud of into customer use.
A downside I see is that testers who never work outside agile projects may never learn to deal with complexity, which concerns me for the future of both AI projects and big software projects. Getting your head around a big picture and thinking strategically aren’t especially required when you only ever deal with small increments of functionality. For the foreseeable future, society will still need some very large software projects, and those projects need fully functioning testers to come in at a senior and leadership level. Where are testers going to learn the requisite big-project skills?
At the moment I know only one tester who is deep into AI projects: Bill Matthews. Matthews warns that testers will need to develop their capacity for thinking about and handling complexity, and again I see that as a potential issue where agile projects are the training ground for most testers.
InfoQ: Has there been a change in the value that testers are expected to deliver?
Charles: Yes. But as I said above, practices and expectations changed and then they changed again. More than once. Likely they will go on changing. We can hope that most of the outcomes will be healthy, but some will probably be detrimental.
At the moment, many testers get to be quality advocates on their teams, themselves testing but also coaching programmers to improve their testing. Where that paradigm works, I see it as very positive. The increased push to automating unit and service-level tests is also positive.
It’s less positive when managers or programmers think they don’t need skilled testers because the programmers or automated tests can do it all. This is an ancient struggle by now and not worth dwelling on here, though it is important to say that it depends. If you meet the right set of criteria, primarily posing low to no impact to stakeholders from bugs in your system, not having testers could be perfectly okay. But that’s not always going to be the case.
The trend to faster and more frequent releases that allow less testing time for integrated functionality is also both a change and a major challenge for testers of systems where bugs could do serious harm.
InfoQ: Which skills do you consider to be essential for testers?
Charles: That could be a very large question or a small one. I’ll say at the outset that testers need more than skills. To perform well as a tester you also need to develop a number of personal strengths or attributes of character, such as courage and good judgement.
I’ve published a big list of essential skills and personal qualities for testers in a mindmap on my blog: Some Skills & Personal Qualities That a Tester Can Benefit from. The skills and attributes in the mindmap are to some extent contextual. Testers may not need to call on all of them all of the time, but most of us will need each one at some point in our careers. Since publishing it, I’ve added to that list. I’ll use examples from it in my keynote.
For a shorter answer, I’d point to the abilities to think and to learn. Thinking is the stem cell skill for a tester, just as the human brain is the central testing tool. Of course, thinking is a huge topic, encompassing logic, creativity, critical thinking, analysis, synthesis, problem solving… and many other cognitive processes that testers need to apply. But your thinking will be limited if you can’t also learn and keep on learning. And you won’t learn if you aren’t curious—which brings us back to personal qualities and strengths.
InfoQ: What can testers do to acquire them?
Charles: Practice. Cultivate curiosity. As with any other skill, the more you think and the more you learn in varied subjects and contexts, the more you increase your capacity for thinking and learning.
Expand your idea of contexts you will work in, and consider working in one you don’t think you’ll like. If you’ve only worked in agile, try a so-called "waterfall" project, and vice versa.
Similarly, work with people whose professional ideas and opinions—about testing and software development—you don’t think you share. You can learn much that is useful from the "other side", and from defending and implementing your own practices in an environment where they are novel.
Finally, grow your capacity for dealing with complexity by thinking about and understanding complex systems and subjects.
InfoQ: What will the future bring for testers?
Charles: I’m not a prophet so my answer is pretty much the same every time I’m asked this question. The future will bring some probably unknowable combination of social, economic and technological changes, to which we as testers will bring our skills and personal strengths.
We can only control what we have to offer, so that is where I believe we must put our efforts. Testers need to be flexible and adaptable, always learning new skills and methods, and willing to take on new roles and activities.
Nothing new about that!
InfoQ is covering the European Testing Conference 2017 with Q&As, summaries and articles. Earlier InfoQ published a Q&A with Adrian Bolboacă on writing good unit tests.