There is an evolution going on in testing. It used to be that testing was about confirming to the specification. Testers were often brought in too late and had too little influence, but that is changing now as Cirilio Wortel explained in his talk on the evolution of software testing – from insurance to influence given at the Agile Testing Day Netherlands 2015. Testers used to be hired to verify that software complied to the specifications, regardless if those specifications where good or bad. Testers were rewarded by the number of bugs they found.
Cirilo talked about the project for the Dutch UWV in which he worked as a tester. Just by looking at the specifications it was clear that the software could never work, the system they were developing could never go live. The project was over-promising and parts of the specifications were clearly wrong. It was a frustrating situation because the testers were not allowed to question any functional logic, they were restricted to look at it from a testing perspective only. Solutions were brought into the project to solve the problems, but with the main target being to build conform specification, none of these would help. All attempts to solve the problems were focused on symptoms rather than on a solution. Testers were uninvolved and the only thing remotely rewarding was to break the system and to file as many bugs as possible.
Nowadays testers are being more and more involved in both the requirements process as well as during the development of the software itself. Testing is no longer a separate task, the testers support business analysts, architects and programmers by bringing in the testing perspective. New requirements techniques evolved that focus on "what" needs to be build, rather than "how" to build it. With this the focus shifted from complying to specifications to refining specifications and thus getting testers more influence on the actual product being created.
Due to the maturing of software development and the introduction of Continuous Delivery new tools and techniques are introduced to do more real time analysis. New features are specified containing examples of expected behavior accompanied by measurable results. These expectations can now be verified during the development lifecycle, but also in production. Technical aspects can be monitored. A/B testing can be used to compare different solutions to the same problem and functional requirements can be validated with usage statistics.
InfoQ interviewed Wortel about his view of the ongoing evolution in testing, what testers can do to get more influence and how testers can prepare themselves for future developments in testing.
InfoQ: You talked about the ongoing evolution in testing. Can you elaborate some of the main changes that are happening?
Wortel: What is important to know is that this is about my evolution of testing, I’m generalizing quite a bit in my presentation.
For me it was always very frustrating in the 2000’s that I could see the freight train running in on us and not being able to turn it or to even slow it down. My influence as a tester was mostly limited to getting something into the bug tracker and that would not be a guarantee that it would actually make any impact. Testing was done late, it was a reactive activity. Most testers were completely uninvolved and did not take any responsibility for the software being created. This in my opinion is for a large part the result of how the whole industry was organised. Huge projects ran without anybody having a clear vision. Due to the way public procurements were drafted the big consultancy firms didn’t have an other option than to squeeze money out of their customers by generating changes and actively steering towards rework, they had no interest in getting things "done". Testers were hired as an insurance policy to cover other peoples asses.
The tester has become more self aware over the years and became more specialized, with standards, tools and techniques. But until agile made an entrance the influence of the tester was limited. Testing has always mostly been identified as a bottleneck, a necessary evil that needs to be dealt with. Many snake-oil solutions were introduced to solve that problem. Armies of unschooled testers, frameworks, brittle test automation, audits, inspections, certification, anything resulting in more consultancy, but hardly ever having a positive effect.
InfoQ: How do these changes impact the role of testers?
Wortel: With the introduction of Agile, for me in 2007, I experienced that my role as a tester became so much more valuable. I could actually make a change, use my critical thinking and domain knowledge to enable teams to deliver value. I experienced the sensation of contributing, rather than only breaking software.
As with any change, for many testers changing to agile software development comes with resistance and uncertainty. I believe that over the years many testers have settled in their role and are not very challenged to take responsibility outside their own field. For years the work was reactive and unnoticed, all of a sudden it is expected that people are proactive and that they cooperate with developers and business stakeholders. Agile testing demands that you can think out-of-the-box and roles are less articulated. The weight shifts from pure test execution to requirements engineering and increasingly test automation becomes more important, thus increasing the need for testers to be more technical and communicational skills are of vital importance.
InfoQ: Can you give some suggestion what testers can do to get more strategic influence?
Wortel: I’ve experienced that if you try to contribute to any part of the delivery process you will earn respect from other stakeholders. I am very curious by nature and will always start by asking questions. Once you understand the way a process works often with a little common sense (and the testing mindset) you can add value. At first people will be hesitant to accept your input, but if you persist, at some point the roles turn and people will start asking for your advise. I have seen this in practice many times, being it in the field of requirements engineering, development, management, operations. Often as a tester your view on things is much broader than most other roles, often stakeholders are so specialized in a specific field or area that they don’t see the whole picture anymore.
InfoQ: And what can testers do to get more involved in operations?
Wortel: At my current project at VNU Vacature Media operations is pretty much embedded in our team. We do continuous delivery and our infrastructure is of vital importance for our process. Every commit goes out to production instantly and the systems is under constant monitoring to ensure its stability. The infrastructure maintained by our Ops guys provides me with valuable tooling to more efficiently do my work. If I see strange behavior during exploratory testing I can immediately trace that back with tools like Logstash and Kibana. With Graphite we can see the effect on performance and memory usage.
Besides these non-functional purposes, the flexibility of our infrastructure provides us with means to do A/B testing and monitor usage. By defining estimations before going live and measuring if those predictions are met gives us valuable feedback about the actual value being created. We are no longer limited to just our software development cycle, but extend our toolset to production and realtime usage.
InfoQ: What do you expect that the future will bring for testing?
Wortel: What is very interesting to see is that I do less and less hands-on testing. It is still very valuable to do exploration testing to experience how the system behaves with actual human interaction, but most bugs are prevented in the requirements phase. By creating realistic examples of expected behavior and creating common understanding among all stakeholders. Automated tests are implemented during development and guide the actual implementation. What I have experienced lately is that the whole idea of having "releases" has vanished with continuous delivery. Our role is still constantly changing and we really have to become more of a collective, combining all skills rather than having pre-defined roles.
InfoQ: How can testers prepare themselves for it?
Wortel: Embrace change and share knowledge. Don’t try to be protective of your role, but ventilate what it is about, try to seek cooperation with all disciplines involved. Make yourself seen and be outspoken, try to find like minded people, within the project you’re in, but also within the community. Expand your knowledge, try constantly to step just that little bit outside your comfort zone.
