2020 State of Testing Report

The 2020 State of Testing report provides insights into the adoption of test techniques, practices, and test automation, and the challenges that testers are facing. It shares results from the 2020 testing survey organized by Joel Montvelisky from PractiTest, and Lalit Bhamare from Tea-Time with Testers.

From the State of Testing 2020 report:

It is very difficult for us to define exactly what we "see" in the report. The best description for it might be a "feeling" of change, maybe even of evolution.

We are seeing many indications reinforcing the increasing collaboration of test and dev, showing how the lines between our teams are getting blurrier with time.

We are also seeing how the responsibility of testers is expanding, and the additional tasks that are being required from us in different areas of the team's tasks and challenges.

We were also able to see some low level changes and expansions in the technologies tested, the technologies used to test, and the technologies that may be relevant in the future.

More importantly, we see how testers have a broader and brighter perspective of their professional future.

InfoQ did an interview with the organizers of the State of Testing survey.

InfoQ: What are the major changes in the 2020 State of Testing report compared to previous reports?

Lalit Bhamare: Keeping our findings from previous years in mind and by looking around what seems to be happening, we modified our question set to get more detailed and helpful information around certain trends.

Other than that I do not think we made any breaking changes.

Joel Montvelisky: We are trying to focus the report a little more. As simple as it sounds, the hardest thing to do is to choose how to remove some of the questions we asked in the past so that we can maybe add one or more points that are worth checking.

For example, in order to understand how much involvement the testing team has in the CI & CD processes, we had to select two questions to remove from our previous year’s survey.

InfoQ: There is a decrease in the percentage of testers who do testing all the time. Can you elaborate on what is happening?

Montvelisky: This is a fascinating and encouraging finding. Testers are expanding the reach of their activities and responsibilities in order to encompass additional aspects within the process.
For some of them, this may be simply to coach and mentor developers on how to perform part of their own testing, and in other places it reaches aspects as far as monitoring quality in production environments and defining processes in order to ensure quality as part of the work of the whole team, instead of finding and patching bugs towards the end of the release.

Bhamare: Totally agree with Joel here. The role of a tester seems to be changing beyond what they have been doing traditionally. Testers are apparently going more in consultation mode to help promote the Whole Team Testing approach or they are using their testing skills and expertise and applying those in other areas of software delivery/support.

InfoQ: What methods do people use to enhance their knowledge and sharpen their testing skills?

Bhamare: Testers appear to be seeking more skilled-based learning and experience-based learning which they are getting through pairing with team members, conferring with other testers, available online options, and community forums.

There is a visible increase in using conferences, webinars, and other online options as a source of learning.

Montvelisky: I like the fact that testers are recurring more Peer Mentoring and Online Resources, and I think both these trends are interrelated as I have seen a number of cases where experienced testers provide online training sessions and then offer to continue with limited one-on-one mentoring if there are interested parties.

InfoQ: What trends do you see in the number of testers in teams?

Montvelisky: This is one of those questions where we are not able to see a clear trend. Numbers were shifting towards smaller teams, but this last year we saw a small jump towards bigger testing teams.

This may be attributed to either a one-time jump based on the sample of answers, or we may be seeing a shift in direction on the “pendulum-like-behaviour” of development teams as they understand they need testers in good enough numbers in order to help teams deliver high-quality products.
We will need to keep tracking this question in years to come.

Bhamare: I agree with Joel here, indeed. I feel it is highly context-dependent. In some contexts, testers are being embedded in teams, and in some cases, more testers are recruited to support testing efforts.

InfoQ: How does the adoption of continuous integration and continuous deployment impact testing?

Bhamare: The way I look at it, it warrants testers (or anybody who tests in the team) to have reliable and optimum automation in testing in place so that the time saved can be used more for finding risks not known yet. And they can find those risks as early in SDLC as possible.

I feel it makes testers critically think about the automation strategy they can have that best suits their context and make it reliable and meaningful. The flip side of it that I see sometimes is that if their automation strategy is not smart enough (or say if their CI/CD infrastructure is lame) testers end up just writing more automation and spending enormous time in just maintaining it for the sake of keeping the pipeline green. These efforts hardly contribute to the user-facing quality of the product and add no meaningful value.

Montvelisky: In my mind they are helping us to be part of the larger Quality Assurance process. The most immediate factor is the inclusion of Unit Tests and the Product Integration activities as part of the testing and quality process - getting testers to be part of the definition and analysis of these activities, something that up to now had been out of reach to our teams.

In the longer run, and specially within organizations working with Continuous Deployment, it means the inclusion of the Testing Team as part of the DevOps group, as we also have eyes, ears, arms and legs in the deployment and monitoring operations of Production Environments.

InfoQ: What testing techniques and methodologies are used?

Montvelisky: The report shows testers continue to use a mix of techniques to perform their work, with Exploratory and Scripted testing being used jointly by the majority of testers.

We also see some practices that are commonly used such as Pair Testing, Bug Hunts, and Review of Production Logs.

No less important in this area is the use of different approaches as well, where we see important numbers of testers mentioning their teams are consciously shifting their testing both left and right, and also implementing more extensive approaches such as MT (Modern Testing) as defined by Alan Page and Brent Jensen in their Modern Testing Approach and Principles.

Bhamare: I don’t see any significant change here as compared to what we already know from previous reports. There is a mix and match of everything that exists but indeed, Exploratory Testing and Session Based Testing continue to be embraced more and more.

Bug hunts and production monitoring seem to be picking up more traction.

It would be interesting to see how the current scenario where most of us are working remotely affects the techniques and methodologies used, if at all changing.

InfoQ: What are the biggest challenges that testers and testing teams are facing? How do they deal with them?

Bhamare: Racing against time and not-getting involved early enough in the process continues to be a top challenge. However, we can also see how challenging it is becoming to create and maintain test data and test environments.

All in all, if we combine smaller challenges in one thing, we can say that time and effort spent on making testing easy is one of the big pain points along with traditional problems like politics, not testers not valuing testing enough and so on.

Montvelisky: For many of us, the most challenging aspects are still timeframes, early involvement, and communicating the value of good and proper testing.
Still, for many testers they have challenges in other areas such as coordinating Offshoring/Outsourcing, Political issues, and time “wasted” on side tasks that are not related to testing such as training other teams on product features, setting up environments for the development team, or reviewing potential issues coming from outside of the development organization.

InfoQ: What practices or technologies will become more important in the testing world?

Montvelisky: My answer here is clear, Data Analysis. As we move towards work on production systems, where data is fuzzier and we are more involved in analysis than discovery, we will need to become experts in Data Analysis in order to make quick decisions based on very large amounts of data flowing from the system.

Bhamare: From the report and personal experience, Data Science, and Analytics testing. AI, ML, and VR are there too but the data field would certainly be a game-changer in my opinion.

InfoQ: What would testers like to see changed to do their job better?

Bhamare: Testers who are embedded in teams as a single dedicated tester seem to be burning out with overload. It seems they would appreciate more testing support or that the entire team supports the quality efforts i.e. whole team quality approach.

Montvelisky: Here I am not seeing something very different than before. We still want recognition for the value of our jobs, more cooperation from the whole team, and specifically participation from developers in the testing process.

About the Interviewees

Lalitkumar Bhamare is currently working with XING SE as senior software test engineer. He is chief editor and co-founder of the popular software testing magazine "Tea-Time with Testers". He is also the co-creator of the State of Testing report project. Bhamare is a skilled exploratory tester, and practitioner of RST and CDT . He is the lead instructor for the BSST Foundation course with Association for Software Testing, USA, and regularly assists James Bach in his RST online class. Contact Bhamare via twitter at @Lalitbhamare / @TtimewidTesters, or visit his personal blog.

Joel Montvelisky is a co-founder and chief solution architect at PractiTest. He has been in testing and QA since 1997, working for companies in Israel, the US and the EU. He is an international speaker and writes on the QA Intelligence blog. Montvelisky is also the founder and chair of the OnlineTestConf, the co-founder of the State of Testing survey and report, and his latest project is the Testing 1on1 podcast with Rob Lambert.