The state of testing 2015 report shares results from this year’s testing survey that was organized by Joel Montvelisky from PractiTest and Lalit Bhamare from Tea-Time with Testers. It provides insights in the adoption of test techniques and practices, test automation, and the challenges that testers are facing.
This is the second time that this survey was done, following up on the state of testing 2013 report. InfoQ did an interview with the organizers of the survey.
InfoQ: What are some of the major changes when comparing the 2015 survey results with the survey from 2013?
Joel Montvelisky: There were changes in the answers that point towards a general feeling of testers being more secure in their jobs. I think we can also see some industry changes pointing towards an even wider adoptions of Agile principles (that are becoming the standard in most organizations) and the testing techniques and methodologies that are required to succeed in agile development environments as a tester (e.g. more automation, leaner testing, more direct communication with devs, etc).
Lalit Bhamare: In comparison with previous results, I see changes on multiple front, majority of them being on "increased self-awareness" and "tester’s mindset". By that I mean testers’ increased awareness towards their job, role, importance in the field and most importantly around their awareness of problem areas that can enable them to function better when solved. These major changes I’m talking about can be realized by considering the survey result as a whole. For example, just have a look on page 14 and 15. Comments over there reflect the progressions testers have made on multiple fronts. Notice the change in attitude towards hiring? Well, that’s a major change I would say. Also notice what tester have to say on what they want to see changed in testing world. That reflects their awareness and mindset. I’m happy to see such changes becoming visible.
InfoQ: The survey mentions that 37% of the testers are involved in requirements gathering. Can you elaborate on the benefits of having them involved?
Lalit Bhamare: From personal experience I can tell that if testers are not involved in requirements gathering they usually end up testing implemented solution of the requirements which can be a wrong or incomplete implementation or implementation of the wrong/incomplete solution at all. That results in wastage of time and money of course. Being involved in requirement gathering enables testers to critically analyze them well before it’s too late & before it becomes a costly affair. How would you like to be informed about risks? "There is a big ditch in our way. Watch out..." way or "Guys, we have fallen in a big ditch"...way? That’s the difference it can make when (or not) testers are involved in requirement gathering.
In my opinion, 37% is still a poor number. But I’m glad that testers are demanding to be involved in req. gathering (see page 15). The real change on this front would come when involving testers in requirements gathering phase does not remain a question anymore.
Joel Montvelisky: I will start by agreeing with Lalit saying out loud that 37% of testers involved in requirements gathering still seems to me like a very low number!
In my mind testers should have a bigger role in the requirement’s gathering process given the fact that we end up serving as the internal customer advocate within the development environment, and as such we need to be more closely linked to our end-user and to the requirements process.
To answer your question, when testers are directly involved in the requirement’s gathering process they are able to grasp not only the "dry requirement" that can be written down, but also the context in which the requirement was formulated. This "seasoning" is crucial when further interpretation is needed, specially when a requirement or a user story can be translated into different implementations of the same functionality.
InfoQ: The survey shows that testers are documenting less. Can you explain this?
Joel Montvelisky: It is important to understand that the number of responders also increased considerably from the first State of Testing survey we ran back in 2013 and the one we ran earlier this year, so maybe we are simply getting a better reading of what is really happening out there.
Still, if we assume that the numbers did not really affect this answer, and that in fact testers are documenting less (while still maintaining the internal proportions of what they document), then this may be because the Development Community as a whole is moving towards a more Agile and Lean environment where documentation is reduced in favor of direct communication.
But I guess this is something that we will want to keep reviewing in our future surveys to check if this a recurring trend and look for the underlying reason behind this.
Lalit Bhamare: I’m pleased to learn that. Traditionally, out of 100% of the time allocated for testing, testers end up using only 30% (roughly) of that time in doing real testing and 70% of that time is wasted on unnecessary, heavy documentation and similar activities which are usually done for compliance sake or for training and audit purpose (see Test Team Challenges on page 13). With the rise of Context Driven Testing practices and Agile/Lean methodologies, I feel that testing groups are realizing the value of spending more time on meaningful testing and doing context appropriate documentation (which is usually lesser than the traditional types). This also implies that testers now know the smart ways to document testing which will eat less of their time.
If I’m allowed to explain this further, organizations and test groups seem to be catching up with the "doing more with less" idea and this is a good sign.
InfoQ: What are the main challenges that test teams are facing? Do you know how they typically deal with them?
Joel Montvelisky: I guess the report makes it very clear that the top most challenges of teams and their managers are around growth and time.
Growth is reflected on the challenge they report in hiring good testers, and in principle this is a good news for the industry as challenges in hiring can be translated into a higher demand for good testers (this is for all of you who thought testing, and manual testing, was a dying profession...).
The second main challenge is around timeframes. Again, I guess this is related to the fact that the world is leaning towards more lean and agile practices, and this approach challenges our previous assumptions of long and comprehensive testing cycles, and places us on a reality where we need to work more based on risk analysis, automation, development testing, and overall less time to test.
Lalit Bhamare: Summing that up in one line, I would say the main challenge is "doing more with less". That means doing more (and meaningful) testing in less time, finding more defects in less time, doing effective/more testing in less budget, doing effective testing with less testers, doing effective testing with less available information and so on.
I’m not sure if majority of testers have found ways to deal with it but among those I know, they are solving such problems by following methodologies like Rapid Software Testing and principles of Context Driven Testing i.e. by becoming a "thinking tester".
InfoQ: Is testing becoming more agile?
Lalit Bhamare: I’m afraid, I can’t confidently say that. That is mainly because what Agile (or agile) means to me or you may be different than what respondents think is agile according to them. For some people doing stand-up meetings makes them Agile, for some of them developing software in sprints is Agile, some say their use of particular tools makes them an Agile team. Some call themselves an Agile tester because they happen to be part of Scrum or Agile team. But do just those things make testing agile? I don’t think so. The better question would be, "Is testing becoming truly agile?" What factors would make testing truly agile?
I feel that our industry needs to come on common conclusion on the meaning of agile, when we refer it in the context of testing at least. I hope, in future surveys we will be able cover it and would be able to see a clearer picture. I must admit, it’s not that easy.
Joel Montvelisky: In my mind page 11 of the Report, where we see that 88% of respondents are working based on Agile Methodologies (up from 78% on the last survey back in 2013), makes this an irrefutable reality: companies are switching more towards Agile. Having said that, I agree with Lalit that many people will define Agile in different ways, and I am not 100% sure this is a bad thing in it self…
But I also believe the world is always more complex than what we like to think, and this is why we let people choose more than one answer to this question. And so, we see that in hand-in-hand to the 88% working Agile or Agile like, we see 42% working Waterfall or Waterfall like 15% working based on their "unique models" and even 6% that don’t really follow any model or principle.
These results can be interpreted in a number of ways, but one that I think is more correct is that most people and organizations like trying things and introducing constant but slow change to the ways they act and work
No less interesting is to see that already today 14% of respondents are already starting to work based on DevOps, and to me this is a forward evolution of Agile, so it will be interesting to keep our eyes open to see if this is a trend that will grow next year.
