2019 State of Testing Report

The State of Testing 2019 report provides insights into the adoption of test techniques, practices, and test automation, and the challenges that testers are facing. It shares results from this year’s testing survey organized by Joel Montvelisky from PractiTest, and Lalit Bhamare from Tea-Time with Testers.

From the State of Testing 2019 report:

"With about 1,000 participants from more than 80 countries, the survey aims to provide the most accurate information of the testing profession and the global testing community. Held yearly, the survey also captures current and future trends."

Some of the conclusions mentioned in the report are:

• Testers still need to be versatile and take up work in additional tasks and realms.
• People keep moving to testing from other jobs or positions; testing is an attractive career that is flourishing in the eyes of the Industry.
• Testing teams are getting smaller year after year.
• Agile keeps being the most relevant development approach; DevOps has passed waterfall and is now second.
• Testers blend different types of testing techniques as part of their work.
• Organizations are shifting their testing, expanding both left and right.
• There is increased collaboration between test and dev; the lines between teams are getting blurrier with time.

InfoQ did an interview with the organizers of the State of Testing survey.

InfoQ: What are the major changes in the 2019 State of Testing report compared to previous reports?

Joel Montvelisky: It surprised me a little to see a slow down on the trend we had where a higher numbers of Junior testers were joining our profession.  It is not that we do not see new testers in the survey, but we see less compared to previous years. Hopefully, this is a one-time point, but we will need to track this in future editions of the survey and report.

We also saw more and more references to "shift-right" practices such as testing in production and involvement in deployment operation.  This is not a change per-se, but a strengthening on this position for testers in the industry.

Lalit Bhamare: To add to what Joel said, even though junior people joining the testing profession sounds like a declining trend, the approximate growth of experienced professionals moving to the testing role from non-testing and non-IT backgrounds sounds like a positive sign to me. That is, people are taking a conscious decision to move to the testing field, apparently because they find it challenging and understand the power that that role offers them to contribute to software quality. I am glad as long as people are joining the testing field, no matter at what level of experience. 

I won’t call this one a major change, but testers taking on other activities (technical or otherwise) beyond writing automation and scripts seems to be picking up morem and that is a promising change, in my opinion; promising because testing for me is not just about investigating a product, but also the system and people and the relationship between these three. Testers’ power of observation and sharp ability to understand subtleties of human/system behaviour makes them more a powerful contributor to keep a "system" away from collapse and explosion. I have discussed some of my arguments in my blog Evaluating and Empowering Testers in Agile Teams. 

InfoQ: Where is testing heading as a profession? How does the future look for testers?

Bhamare: To put it in short, I think "testing" is slowly getting spread thin across the processes, culture, and mindset of teams, regardless of what job-title an individual has. That said, there may not be (I am still just suspecting that) a dedicated "tester" role in teams in the future, but "testing" would certainly have to be part of the job responsibility of everyone in the team, or at least a few individuals who have the skills and mindset to do that. 

Does that mean testing would die as a profession? I believe it would not; it would only transform or evolve and spread its horizon beyond where it is being seen as being so far. And that, in my opinion, would mean the role of a professional tester of today would transform into a quality advocate, facilitator, coach, or enabler in software teams. I would expect that in the future, a skilled tester would be a valuable asset for product teams to guide them in their understanding of risks, ability to see degrading quality, and help them evaluate and understand that, and act on it before it’s too late. 

Montvelisky: I think "the Jury is still out" on this question.

On the one hand, we are seeing the bulk of the testing professionals concentrating mostly on regular testing tasks, working on their automation efforts, understanding requirements and user stories.  

But on the other hand, we start to see how important parts of the testing itself are being done by the none-tester-team-members.  Also, we can see how part of the tasks done by testers are around coaching others how to test, and enabling the testing process by others.   These points resonate very strongly with the ideas of "Modern Testing" being put forward by Alan Page and Brent Jensen on their AB Testing podcast, and what Lalit was writing in his reply above.

InfoQ: What are noteworthy developments in the salaries of testers around the world?

Montvelisky: I do not think this question in particular brings up something we do not already know about the industry. What is encouraging here is to see how professional testers can expect their salaries to go up as they gain experience and tenure in the profession.

Bhamare: That’s a tricky one to answer. When it comes to salaries, there are lots of factors playing a role. But I would say, if you’ve got what it takes to fulfill the job-requirement, organisations seem to be willing to appreciate that with competitive salaries. 

InfoQ: How do testers learn about testing, and how do they develop their testing skills?

Montvelisky:  I think most testers still look for knowledge, mostly around the formal channels, such as books or conferences or courses; and for me, this is actually a pity as the amount of free information available on podcasts, webinars and blogs is actually amazing!  I can attest that personally, this is how I keep up-to-date and how I get new ideas.

Bhamare: I have not seen any breaking change here. As Joel rightly said, it looks like testers continue to rely on the traditional way of learning testing, i.e. on the job training or learning while they practice testing and so on, as a primary way of learning testing.  

I would recommend them to come out and see all the great resources that are available to them. And by that, I do not mean only for testers, but also for engineering team leads, and development directors who are nowadays responsible for testers’ career and growth, who report to them.  

It is important that everyone in an organisation who has any sort of connection with software testers come out and get clarity around what is going on in the testing world, and how they can use these resources to help their testers grow. 

InfoQ: What kinds of testing techniques and methodologies are testers using?

Montvelisky:  Around this topic, I was really happy to see how the adoption of BDD and TDD is increasing.  Also, it was refreshing to see how testers are starting to see their "testing in production" activities as testing activities (and don’t let the name fool you; this is not the regular testing we do on a different environment).

These two trends help me draw a picture for our future, in which we take advantage of the process and technological trends in order to improve the way we approach our quality-driving activities.

Bhamare: I won’t call them testing techniques per se, but yes, the adoption of TDD/ATDD/BDD is noticeably increasing, I suspect in order to be closely connected with automation efforts and frameworks/tools being used though. I would like to see how this adoption is helping them to solve quality related problems and improve the software quality on the ground. Perhaps we could try to figure it out through surveys in the coming years.   

I personally feel pleased to see a remarkable increase in production deployments and logs monitoring activity. I have time and again emphasised the benefits of it through personal experience, and I would like to see this trend catching up more. 

InfoQ: How is agile impacting the way that testing is embedded in teams and organizations?

Montvelisky:  We see more non-testers running testing tasks, as well as more testing tasks being done by non-testers.  This is the "trivial" answer.

The less trivial answer comes from people talking more and more about shifting left, testing in production, BDD, retrospectives, etc., all of these being indicators of deeper work focused on generating quality in other levels and aspects of the SDLC.

Bhamare: I agree with Joel. There is a noticeable increase each year in activities testers are taking up, in addition to their traditional job definition. This does indicate that responsibilities are being intertwined or shared across job roles. 

How that impacts the organisations is difficult to answer in general. Maybe, they are able to save costs and able to expedite the product deliveries, but that all really depends on what organisations understand by "agile". And I won’t go into that topic here :D. 

InfoQ: There was a new question this year about what techniques or processes testers are following or implementing. What are the outcomes?

Montvelisky:  As this is a new question, we do not see trends, only snapshots.  But still, we see 60% of testers saying they are shifting testing left, 45% of testers saying they are testing in production, and even 11% saying they are practicing Chaos Engineering.

These numbers are encouraging, but at least for me, they raise some questions regarding the actual practices testers place under these processes; a question that I plan to suggest for our survey next year.

Regarding new questions like this one, an interesting fact about this project (both the survey and the report) is the way it relies on the community for its definition and distribution.  Each year there is a new committee of world-renowned testers reviewing the questions and suggesting additions to the survey, word about the survey itself is spread via volunteer bloggers and community sites, then the report itself is translated into multiple languages by teams of volunteers who want to help spread the word in their local communities and languages. 

Bhamare: The data is there in the survey, as one can see, but to decide about the outcome or draw meaningful conclusions out of it, indeed we should wait and try to understand it better over the coming years. 

InfoQ: What trends do you see when it comes to the challenges that test teams are facing?

Bhamare: I don’t see any remarkable difference from how things have been so far, when it comes to challenges faced by testers/teams. Early involvement of testers in the development process still looks like a challenge, and that is unfortunate. Shrinking team sizes and lack of training support or lack of whole team testing at the same time sounds like a truly challenging situation. 

Concerning most of these challenges, I do suspect a connection between the increasing adoption of agile/DevOps, and that testers are reporting to engineering teams, where not enough people have experience with professional testing or know how to really best utilise testers in the team, beyond asking them to write automation. 

Until we make "testing" education a part of everybody’s learning curriculum, these challenges are unlikely to fade away. I hope that when organisations finally gain clarity on what they really expect from testers and how they can best utilise their abilities at work, things will get better. I hope for the best! 

Montvelisky: I agree with Lalit here, on the fact that we see more or less stability on the challenges we are encountering as a profession.  But for me, there is dissonance, as we are not seeing many challenges related to the adoption of Agile and DevOps, and this got me thinking.

These answers are not in the survey, but for me, the explanation is that most testers still see their work on areas such as the deployment process, production environments, monitoring, etc. as tasks and challenges that are "outside of their testing work".  This is not a good thing, as the moment we do not see it as part of our work, we approach it with less legitimacy and less sense of ownership…  

InfoQ: What technologies or subjects do you expect to become more important in the future of testing?

Bhamare: I believe that technical testing (automation and other things) would gain more importance, not only in terms of adoption, but also in terms "invention, analysis and investigations". That is, all this tool-assisted and technically-facilitated testing would have to undergo critical evaluation for the value it is adding to product teams and quality, beyond making testing(?) cheaper and faster. Let’s not forget those Cheaper, Faster and Quality equations. In the future, I feel the "Quality" aspect will start weighing in more again, putting pressure on the other two aspects to adapt, cope with, or evolve. 

Montvelisky: Looking at the other side of the coin Lalit mentioned here, I think that as we work more on production and expand the definition of the value provided by the QA Team into production and into the actual work of users, we will see more technologies around monitoring, data analysis, and even experimentation, such as A/B testing on apps to take more of an active role in the toolset used by testers.

These are at least my hopes, as I see a time when we actually become Quality Engineers, expanding on our traditional role of Testers within the team.

About the Interviewees

Joel Montvelisky is co-founder and chief solution architect at PractiTest. Montvelisky has been in testing and QA since 1997, working for companies in Israel, the US and the EU. He is also a blogger with the QA Intelligence Blog, as well as the founder and chair of the OnlineTestConf, the co-founder of the State of Testing project, and in his latest project he releases a the Testing 1on1 Podcast with Rob Lambert.  

Lalitkumar Bhamare is currently working with XING SE as senior software test engineer. He is chief editor and co-founder of the popular software testing magazine "Tea-time with Testers". He is also the co-creator of the State of Testing report project. Bhamare is a skilled exploratory tester, and practitioner of RST and CDT . He is the lead instructor for the BSST Foundation course with Association for Software Testing, USA, and regularly assists James Bach in his RST online class. Contact Bhamare via twitter at @Lalitbhamare / @TtimewidTesters, or visit his personal blog.