Agile methods have many traditional test management activities built into them. With desired agile team traits like self-organising, role blurring and skill diversification, the nature of test management is changing. We have to question whether the role of Test Manager should exist in effective agile organisations and how the activities which have long made up the role are divested?
According to Tom Roden and Ben Williams agile test management is about establishing the right culture of testing, educating people and getting hands-on in demonstrating how teams should go about making testing support agility. However, these are not traditional test management traits, which lends weight to the thesis that the role of test manager will cease to exist in agile organisations.
At the Agile Testing Day Netherlands 2015 Roden and Williams talked about the changing face of test management. They showed how the role of test managers will transform into one that enables teams, rather than managing them. The resultant shift in approach and responsibilities will also mean that agile teams will pick up many traditional test management activities. which changes the role of the test manager significantly. Further more test strategies in agile should be treated as living documentation, that live, breathe and grow along with the teams and people who use them.
InfoQ interviewed Roden and Williams about the role of the test manager, what makes agile test management important and the value that it can bring, agile test management activities, what test managers can do to prepare themselves for agile and the difference in testing strategy for test managers in agile.
InfoQ: Agile methods like scrum do not mention a specific test manager role where waterfall projects often included it. In your opinion is that a problem?
Roden and Williams: We might, somewhat mischievously, ask who is it a problem for? If we took the hard line view that all management is waste, from an organisational perspective we’d have to ask what value it serves to have a manager specifically for testing. Is that useful or wasteful, or more pragmatically, does the benefit from having one outweigh the cost of this overhead (temporary waste), until a time when they are no longer needed?
From the perspective of someone who is a Test Manager, over the long term it might become a problem if we agree that the role is no longer needed, or that many fewer Test Manager roles are needed. In the short term the problem is probably more about expectations for Test Managers who find themselves working in agile environments. In companies faithfully practicing agile methods, as a minimum, they will find that the type of work they are needed for is radically different than the work they were familiar with in traditional software projects.
Scrum doesn’t have a Test Manager role, but nor does it have a Project Management role, largely leaving that set of responsibilities down to the Product Owner (as if they don’t have enough to do, what with ‘owning the product’ and all). In practice, particularly for large programmes in large organisations, there are often project, programme and portfolio managers that bridge this gap and help Scrum scale..
So the question to answer is whether organisations practicing agile have a need for someone outside of the feature teams, to work in a testing related capacity? And if they do what is this role, when is it applicable / useful, what scope and scale does it have?
InfoQ: Can you share your high level view of agile test management?
Roden and Williams: Our view is that Agile Test Manager is a misnomer. Testers in agile teams do not need managing per se, so the title ‘Manager’ should be dropped altogether. Those teams will also perform a large chunk of the test management responsibilities traditionally associated with the Test Manager role, note the difference between test management and test manager
We should separate the role from the function. The role may not be exactly clear to many, which may be a result of a misunderstanding in the changes in testing that agile methods bring about. How much a team can realistically be expected to pick up and run with effectively and over what timescale is one interesting challenge. The vast array of different implementations of ‘agile’ methods and it’s hybridisation is another, that makes any specific role definition difficult to assert.
With the caveats out of the way, our view is that this sort of role works across many teams, championing, curating and coaching testing as a function, whilst instilling a mindset and toolkit of practices and techniques. It is about establishing the right culture of testing, educating people and getting hands-on in demonstrating how teams should go about making testing support agility.
What it is not, is writing long winded shelfware test strategies, heavy duty test plan documents, test schedules, holding lengthy bug triage meetings every day and asking testers for status reports every few minutes. If you are doing this and working in an agile environment, then you aren’t working in an agile environment or at least you won’t be there much longer. That might sound a bit harsh, intentionally so. Not that the work of Test Managers in traditional projects isn’t very useful to making those methods work, it is and it can be very challenging. Rather, in an ‘agile’ context those things are just not the best use of anyone’s, including, very bright Test Managers’ time!!
InfoQ: You mentioned that test managers can also act as a coaches. Can you give some examples of this?
Roden: This is very much the route I took in my career. Although I actually grew up using exploratory testing and practices akin to acceptance test driven development, I spent many years in traditional projects.
I had the good luck of being pitched into helping out some agile projects that were struggling with testing. With the help of an excellent couple of mentors myself, I found that what those teams needed was coaching not managing. Trying to test more in less time, focusing the attention of testing on the right risks and harnessing the right skills takes some radical shifts of mindset. Because it is a behavioural shift it isn’t about performing planning, allocation and tracking for teams. In fact that is directly opposed to the desired effects of empowerment and self-organisation.
Roden and Williams: A major premise of agile is enabling self-organised, fully cross-functional teams and so the goal of anyone outside the team should be to support that purpose. This is very congruent with having advisory, coaching functions to provide that support.
Those test managers that find themselves implementing and coaching agile testing may find that they start to get at least as much of a kick out of coaching as anything else. We have seen first hand many others take on various coaching roles. Some have made a direct shift into scrum mastery or agile coaching, while retaining testing as their primary area of expertise.
There are also still senior roles in testing that have a more coach oriented slant. Maybe not the head of testing roles which tend to have more accountability, but roles exist which are about coaching good agile test practices, usually across several teams. The questions still remains whether these are transitory roles, needed until teams and organisations are completely self-sufficient. At the point of self-sufficiency, the role would either move to other teams, morph into something else or evaporate.
InfoQ: What makes test management important in agile? What value can it bring?
Roden and Williams: The value is in enabling teams to bring confidence in the quality of products in shorter and shorter time periods. Testing equals information provision about the various risks in using a solution. Finding enough of the right sort of information in a tight timebox is challenging and requires many different types and levels of testing to be thought about and completed.
For example, using an appropriate blend manual / exploratory and automated test approaches is not an easy decision to arrive at. Supporting teams in selecting the right level to implement automated tests, whilst also pairing with them to devise and execute a session based test charter will be invaluable in helping teams of multi-disciplined members make the shift to an agile testing mindset.
More than this, the practices born through driving delivery with testing (Specification by example or Behaviour Driven Development) make the value of testing not just about ‘have we built the solution to work as we expected’ but also to try and ‘build the right product in the right way’ from the outset.
Specification by example (for example) teaches us to specify very clearly in advance of building anything, what exactly that thing needs to do in order to be deemed acceptable. The value this brings is in shared understanding, cutting out ambiguity that leads to rework later on and to untold discussions about what the feature should be doing.
Being able to teach and coach practices like this throughout an organisation is incredibly valuable amplification, in order to get the most amount of useful information out for the minimum effort spent.
These are just a couple of specific examples. More generally, anything that Test Managers can do to enable teams to devise appropriate testing solutions, skills and practices will be beneficial. When this is multiplied by covering a larger number of teams, then the benefit is amplified.
InfoQ: Which kinds of test management activities can be done in agile? Who would be doing them?
Roden and Williams: Most test management activities still need to be done in some form in an agile world, especially if we consider management of the tests, rather than the testers. The nature, format and timing of many of those activities needs to change significantly though.
We have run an exercise a few times with large groups of between 20-40 people, to check what the popular opinion is in answer to the very question you ask. Participants list out all of the tasks involved in traditional test management. We then asked them to assign each task to one of several areas, one marked ‘don’t need this in agile’, one marked ‘should be owned by the feature team’, one marked ‘should be owned outside the team’ and we also later introduced a section marked ‘might need a kick from outside the team’ where some outside stimulus may be needed to support teams initially.
The results are always interesting. Unsurprisingly there is some disagreement between different people (or rather roles they are filling). For example, feature team members may challenge some of the things that Test Managers think need to be owned outside of the team. However, what does come out every time is the realisation from members of teams that they have to pick up lots more activities than they had previously thought. There are also a fair few activities which, if left purely to the team, could result in some quite expensive decisions from an organisational perspective.
Activities like designing, automating, executing, maintaining and scheduling tests are fairly uncontentiously filed as ‘owned by the team’. For activities like tooling, training, mentoring, career development, budgets, hiring, appraisals, even strategy, the model of ownership is much less clear.
InfoQ: Why is it so difficult to allocate activities like tooling, training, etc?
Roden and Williams: Often teams that become more self-sufficient will start to take control of and explore options, for example getting the most appropriate tooling and training there is. This is a great thing to have happen.
However, when we work at some scale, within a department or organisation, locally optimising for one team and set of circumstances, at the cost of other things, may not be an equitable trade off. So, if we end up with a thousand different tools and technologies to curate licenses for, versions for and most importantly skills alignment for, this may not be a wise choice for the overall organisation.
Some centralised decision making can be beneficial to the overall organisation, particularly when paying consideration for something that serves everyone well whilst allowing for the exceptions that prove every rule and being flexible to some needs. By this we mean don’t mandate that everyone has to use exactly the same tool, just because it sounds like a similar job. Where there are legitimate reasons for varying, consider these, but vary them based on sound economic reasoning, not just magpie syndrome (following the desire to chase new and shiny pieces of technology).
InfoQ: What would the difference in testing strategy be for test managers in agile?
Roden and Williams: Firstly it is worth stating when we say test strategy we usually refer to the long lived approach to testing that a given team or department in a given context adopt. It covers their principles, processes, ways of designing and implementing types and techniques of testing, automation, risk coverage, view on quality and other similar things.
In previous times, test strategies were quite often weighty tomes used to capture everything a test manager knew about testing, their magnum opus. After creation they would be put on display for a bit before being placed carefully on the shelf and forgotten about.
Test strategies, even documented ones, are still very relevant in agile teams. They often require very broad and deep knowledge of testing to drive good behaviours, which is the domain of the test manager. The trick is to treat them as living documentation, that live, breathe and grow along with the teams and people who use them. This means keeping them concise, usable, appropriate and written in a format and medium that encourages regular use, extension and amendment as ways of working change.
One example of this is below for using agile testing principles. Here are a set of principles to convey our thoughts on testing, they are common enough to be applied across multiple teams and the details underneath can be moulded to fit each team’s context.
InfoQ; Can you give an example of using a test strategy with an agile team? How did they do it, and which benefits did they get out of it?
Roden and Williams: The above principles evolved from a simple test strategy. It was more of an extension of some core team ways of working, a few wiki pages and documentation within our test automation framework. Taking a lightweight strategy can take just a couple of hours of team communication to agree and then it can evolve as living documentation.
For example, one team we were working with used a few simple principles like done means acceptance tested, collective test ownership and empirical measurements. The team measured a baseline amount of effort in regression testing, production bugs and productivity, as well as less tangible things like stakeholder confidence. The culture existed that encouraged the team to own the whole delivery stream, so these principles within the strategy had profound consequences.
Adopting automation under the UI, expressed in a common domain language (BDD), measuring the change in these metrics and getting everyone responsible for adding to and maintaining the tests created had a massive impact. Other factors like pairing, test design principles and better collaborative story discussion using specification by example were added to the strategy over time and so the understanding and discipline grew, but as living, breathing practices. This is missing in so many test strategies produced.
The evidence from this project is below, going from 150 bugs in its first release (delivered two months later than the expected eight months) with an increasingly cumbersome regression cost, you can see the impact of deploying an agile test strategy towards the end of the 2nd period:
InfoQ; What can test managers do to prepare themselves for agile and adapt to the changes that agile brings?
Roden and Williams: For those test managers that want to make the move over into agile environments (and when we ask this question these days it seems this is the overwhelming majority), there are a few things to figure out. Skills picked up in test management, such as communicating information in different ways to different people, may lend themselves to coaching or scrum mastery roles, away from a specific and dedicated testing role.
In order to support teams in getting to grips with successful agile testing, test managers will need to learn the massive differences for themselves. While the skills in testing, the need for succinct information provision and risk management are still very relevant, the application of them will be very different.
Spend time understanding what activities are still valuable and how to support feature teams in taking ownership of some of them (with or without dedicated testers). This will help test managers avoid the pitfall of trying to hang on too long to tasks that are not needed or those that inhibit the independence and self-organisation of agile teams.
Finding the activities where teams still need support and specialist knowledge and enabling them to learn and develop those skills will be how the most successful test managers evolve into more useful organisational change agents and enablers.
About the Interviewees
Ben Williams is a coach, consultant, transformation specialist and author of ‘Fifty Quick Ideas to Improve your Retrospectives’, he helps organizations deliver real business value, consistently and faster. Applying a wide range of tools, techniques, and experiences, Ben coaches teams and leaders to appraise and refine their work systems. Drawing predominantly from agile and lean disciplines like Scrum and kanban, Ben has been involved in driving some radical and large scale agile transitions, working as a coach and servant leader. He is a regular speaker within the UK and internationally and was rated top track talk at Agile Testing Days 2014. Follow Ben on twitter @13enWilliams
Tom Roden is a software delivery coach, consultant, quality enthusiast and author of ‘Fifty Quick Ideas to Improve your Retrospectives’ and 'Fifty Quick Ideas to Improve your Tests'. Tom works with teams and people to make the changes they need to thrive and adapt to the changing demands of their environment. He collaborates with those intent on delivering high quality software with speed and reliability, supporting ongoing improvement through process and practice refinement, influenced by agile and lean principles. Tom specialises in agile transformation and quality, from management and strategy to practitioner approaches and techniques that help teams and organisations continuously improve. He is a regular speaker within the UK and internationally and was rated top track talk at Agile Testing Days 2014.