Good, thank you.
Todd: To get us started, maybe just tell us a little bit about yourself.
I’m an Agile coach, I’ve been playing around in the Agile sphere, actually, before Agile was even coined, before a couple of guys met on a mountain top and sorted it out, very early into XP, in Australia I was doing XP down in the skunkworks of the organization and I’ve been working with large organizations ever since in their Agile transformation.
2. What brings you to Agile 2013 this year?
For a number of reasons, I am here like you, on behalf of InfoQ to do some interviews which is a lot of fun, here as well I am part of the Agile Alliance consultant board, I guess for one of the better terms, occasionally the Agile Alliance will ask members of the constituent different questions, so as a group of those people I’ve been invited along in that, also speaking here at the conference and also doing some podcasts as well while I’m here. So, lots of things keeping me busy.
Todd: Excellent. So, you had a talk called the 7 Deadly Sins of Agile Test Automation.
That’s right.
Todd: That’s an amazing title, first off.
I did notice a few more people have actually used that title as well.
Todd: Perhaps, they had a different take, so, tell us a little bit about the session and what are the 7 ones, as well.
As you know, there are the 7 deadly sins and this came as a joke of mine, from a course from a long way back, of course a movie in the 1990s, but also as a book before that, but really it was something that my colleague, Adrian Smith, and I, as we were doing a program called Building Quality In for a large organization in Australia, we realized that organizations understood testing and why it was important, but couldn’t really put their head around where things were going wrong, so we tried to group them together and there were about seven of them, so that’s how we came up with the seven deadly sins. So, you’ve got things like envy, envy is where you have management that compare manual and automation, but it’s really a flawed comparison, they think automation is important and they want to get rid of manual, but actually you need both of them. Then you’ve got gluttony, which is you overindulge in commercial testing tools. A lot of people were finding they have these big monolithic tools and they were trying to bend them to do everything, but even worse they weren’t expending the effort into either training or giving them to the entire team, so there is this very much isolated set of users that were using the tools, which doesn’t really go very well with our whole team approach in Agile.
And we have lust, which is where testers really love the user interface and don’t really test any other level and as we know user interface tests are very brittle, so trying to teach people to get away from user interface and actually spend a lot more of their time lower down the chain, particularly unit testing, but also interface and acceptance testing level; pride, which is all about collaboration, being too proud to collaborate, so the whole pride thing is reminding people that having testers and developers collaborating together is extremely important. Sloth, which is where you’re too lazy to fix your tests that are broken, particularly when you get a large test suite, when things break down, make sure you go back and keep them maintained, so you have a healthy system. Rage, which is all about the rage of slow and brittle tests and what you do with those to make sure they stay maintained, and then finally greed, which is really about trying to cut costs through automation. So, what we found was those patterns were things we could actually share with teams and put them into one or two patterns and throughout the talk what I really gave was how do you identify them and then more importantly some of the things you can do actually get yourself out of there and what success in those particular deadly sins looks like.
Well, nothing is wrong with wanting to cut costs and tests, but when you think about something like greed, often money is the driver, when really quality should be the driver. So, it’s ok to look at automation, but automation shouldn’t be “ok, we can now get rid of all our QA staff” or “let’s free the developers up just to develop and the QA will take care of itself”. So, sometimes as you move down the path of automation, there is this misconception that it means that things will be cheaper, it’s really you moving things around saying “let’s invest more effort in automation over here so I can free these QA staff up to do the things that are actually really important, which are things like exploratory testing and the like”.
What I found is actually most of the time organizations that are in this scenario actually know they’ve got a problem, they are either trying to look to outsource QA, they think the testing is costing too much, they don’t know what the cost or benefit is always, so typically I found they know they’ve got a problem, what usually they don’t know is how to dig themselves out of it. The thing that drove this, our Building Quality In initiative at a large Australian bank and insurance company, was driven by the fact that they actually moved down the Agile path, but hadn’t taken testing properly along for the ride, and they did a massive Agile maturity survey and the number one thing that kept coming up from everybody in the survey was that testing was not being done effectively. When the management got together, they actually wrote this up on the white board, I’ve got this on one of my slides where they said, “we need to fix testing”, they actually said “we want to fix test automation”, no, what you actually need to fix is testing, test automation is just a part of it and that’s one of the things we talk about in this talk is yes, test automation is important because Agile is all about iterative, is a series of loops, we need to have regression, we need to make sure we have confidence, but actually we still need manual testers.
My usual saying is that testers are a special breed of people, you and I, software developers, project managers, coaches, if we are going to cross the street, we look right, we look left and make sure a car isn’t coming, a tester, someone who is really a tester is going to walk out in the street and not only look right and left, but look out for a piano falling down from the sky because they are looking for those things that are outside the normal. So, really, when you’re getting started in an organization, sometimes people don’t see the value of testers because all they do is this mundane, repeatable stuff and they can’t get themselves out of it and therefore, that then has a flawed effect because they don’t spend a lot of time in training, they don’t spend a lot of time up-skilling, it leads to then “we have to hire more QAs”, it’s just a vicious cycle as you move through. So, as you try to move in Agile, the testers, the developers and the business analysts, all into one group, you have to really rethink about how you approach testing in the first place.
They are all pretty common, depends where you are on the journey, some of them are very early on, like commercial test tools, usually is for people that haven’t made that leap, particularly if they are new at things like continuous integration or they haven’t really invested in ALM, it's something we were seeing very early on. I was speaking to a bunch of people here and they are still in the same situation, particularly when the test organization is still separated from the development team. As you move through, things like pride, collaboration with the developers, only really comes when you've got passed that and you are really talking about collaborating inside of a team, and the test folk actually maybe need a little more skills, the QAs they haven’t had that, so talking about how they collaborate and making them a little more at ease in talking with the team.
Then you finally get to some of the other ones, like brittle tests and the like, are more when you actually have an automated test suite, but then you try to figure out how to maintain it, so you kind of move through the cycle as you go along, at the start you try to figure out “hey, I haven’t got that much automation or the automation I’ve got is not that great” to then “how do I make this whole work in an Agile team”, to finally “I have this set of tests, what do I do with them”, which is when you move on to things like acceptance test driven development, BDD, behaviour driven development, specification by example. And specification by example is an interesting one, because once you move there you start to actually bring in the business analysts and you also bring in the product owner to say “what actual documentation is important to us, what narratives are important to us from the business analysts perspective, let’s actually make sure we are testing them as we go”.
The signal that actually really pushed us forward here was that as more teams were going Agile, there were more requests for QA, because people were trying to figure out this magic number how many testers equate to how many developers in the team, which is a flawed calculation at the best of times, some teams don’t need any QA staff and some teams need more QA staff than software engineers. But in an organization where they only had 100 QA engineers total, they were asking for another 100 more and that was kind of the red light that went off, “really, is this going to continue to build as we do the Agile transformation, are we going to have all these QAs?”, but then the second problem was you can’t actually find skilled QAs easily, anyway. It’s hard enough to find developers, but QAs, particularly good ones are really hard. So, some of those things are usually a bit of a signal that things aren’t going maybe as well as you expected and you need to think about how you want to move forward.
It’s interesting because we often go through Agile transformations, we think about the process, we often think about the software engineers, particularly if we are building software, talk about TDD and pair programming and continuous integration and all those things come out in the wash, but we tend to often leave the business analysts out of the equation, we give them these story cards and we say “go write those” and we don’t give them much more to work with and the same with the testers, if we are going to do some test driven development, that has a massive effect on them because the amount of energy and time moves, for the testers it moves more to the front and in line with the development team and for business analysts it’s the same thing, less of their time at the front and more of it moves alongside where you are in the sprint or the iteration. I think a lot of the books and a lot of the trainers don’t give that the attention it deserves.
First of all you have to understand what the definition of quality is and that’s something a lot of people don’t understand and a lot of lift off or inception parts of the project quite often they’ll say look at the sliders or look at the success criteria for a project and will look at things like time and cost and scope and typically quality is there as well, and out of those four I would always see that quality comes in at number three, nobody put quality at number four, because nobody wants to be seen to be not liking quality, it’s like not liking puppies, it’s quality, something that you like to be seen, but nobody really wants to put it at top because time and cost are usually more important. And a lot of that stems from the fact that no one has really had the conversation what quality really means. If I am building something internally, that it’s going to be just shipping around packages from one side of the business to another, maybe I don’t need to be spending a lot of time on that, but if I am paying my employees, maybe I do care about that because it has an effect on morale.
It’s the same thing if you’re going externally, if you’re an organization that publishes cat pictures on the Internet, maybe that’s not as important as someone who is doing things to heart monitors that are inside of you or maybe even financial transactions. So you have to have that quality discussion and you have to understand what the product owner or what the business owner of the product actually thinks quality is. Because if you ask the team, they’ll have different opinions, the project manager will think quality is getting the project done on time, the developer will think it’s the maintenance of the software, the tester will be thinking that it’s the goal of test passing, but actually it’s none of those things, it’s things like how fast or how performant is the application, is it robust, is it maintainable, all those things need to be discovered, the non-functional requirements and in actual fact, most of them are just assumed, particularly by the person paying the bills, I assume that every transaction is going to be quick, I assume I’ll be able to pick up my iPhone and it’ll be in the app store, I assume that the icon will be pretty, I assume that the color will be red and match the color of our branding logo, and actually we don’t necessarily ask all those questions, they come back to bite us later on, and putting quality in, you can’t build it in later, it’s not an optional requirement, if it’s not there you will have to go through a lot of effort to be able to fix those problems.
Todd: What is the value of building quality in and doing it to the front, how does that manifest itself in value for the company, actually, let’s start with that.
Value for the company, a lot of it really depends on what image you are trying to portray, the old adage of Apple is that you ask a bunch of people is this a quality product and most people will say yes, because they spent the time to make it contour to your hand and be the right size and be an experience, it may not be the flashiest phone on the market anymore, but it’s still reliable, and most people, even if they’ve moved to other devices, still look at it as being a quality item. It’s the same if someone drives a BMW, for example, that brand makes good quality to them, or think about other things in your life, your dishwasher, when was the last time you really thought about that, but when you actually come back to software, it’s not a question that we ask so much. So, it really has a lot to do with the proposition of what you are trying to build and we just have to get the conversation out there. If it’s not important, that’s great, let’s just go and hack in and build away, but there is this assumption that things will actually work and anything we’re adding into the equation, we have to build in performance to make sure it’s performant, we may need to put extra hardware, extra software, we may need to put extra things in place to make that happen and it all costs money, so I think it actually makes a change that people assume things, because it hasn’t been built into the estimate already for the project. If you actually have that discussion, you’ll actually see that there is more of a time difference perhaps, but you need to have that conversation up front, so if you are going to build this gold plated system, it’s going to cost you gold plated money.
Todd: One of the other things you’ve been working with is Agile outside of the IT department, say a little bit about that.
Yes. I actually spoke at a conference back in Chicago in 2009 or 2010, one of those years, about this, very early on I taking the idea of XP practices and applying them outside of IT. So, to give an example, one of the things I found was that a lot of business tasks, I’m talking about business projects where they are doing things like process or procedures or writing manuals or those types of things, people didn’t know what the definition of done was, so I’d work with these teams, I’d write cards on the wall but they would just kind of live forever, so trying to get them back to our usual Agile, we’ve got a three-point card, this is going to take us no more than a couple of days, how do we know we’re done? And getting people to think about that actually is quite hard, how do you know when you’re done your washing at home? Well, you know because the washing machine is finished, and I hang the clothes out on the clotheshorse and I brought them in and they dry and I fold them, that’s when that process is done, there is a clear definition. But what if I’m writing a manual, how do I know it’s done? So, I talked to these people and I said how do you know you’re done, and they go “well, it’ll be in the manual”, “yes, but what else?”, and then things will come out, “well, I suppose when you talk to the manager” and “I suppose you talk to this person”, “oh, we better do some research on that”, actually the tasks that need to be done would come out, it would then become a checklist that people could sign off, that’s really test driven development. Pair programming is another one, just the whole thing of dragging someone to your machine, get them to sit there and go “take a look at this”, code review if you’re writing a document sending something for sign off, instead of sending it and putting it in a pile of stuff, maybe treat it more like a code review and get them to look at it, or even better you do pair programming to get to the end of it. So, taking these XP practices and applying them, the whole idea of not just sprints and iterations, but all the other technical practices, just gave something to build on, particularly for people outside of IT. And they can really see these things because a lot of times in Agile organizations they’ve worked with the IT department anyway, so it’s just taking the same things they’ve seen and applying them to their context.
Honestly, I’ve probably worked in most parts of the organization now and the way you roll it out is a little bit differently. In areas where they do things like policies and procedures and those types of things, it works really well because it’s still doing work, I’ve seen it apply to areas like procurement or to areas like real estate, a little bit different if you are managing things like moving employees from one end of the building to another on a regular basis or you’re building new branches, that’s a little bit harder, but still ultimately the same type of ideas, just a little bit different in the way that you process it, areas like that actually thought that work better with Kanban than they do with the more iterative approach. In fact one of those ones we found that expedites were extremely important, because the managers were just constantly coming along “here is something else, here is something else”, just making the art of everything that’s going on visible, is actually extremely useful in areas like that. And then you go to areas that are still in IT, but outside of software development, people who build databases or people that look out after the security of systems, and again just that visibility of exactly what’s going on is important.
Todd: One of the other things you mentioned to me before was talking about the role of the Scrum master, you had some thoughts on that.
Yes, I recently gave a talk called Scrum Masters: The Full-Time Role Conundrum, really because I’ve walked around conferences like this and I keep asking people is the Scrum master, is the iteration manager a fulltime role? And I get weird and different answers, most people kind of go “yes, but”, I think it’s one of those things that is misunderstood because when you apply that to a lot of organizations, they read it as we have to put something in HR as a Scrum master and that person must be doing Scrum master work 100% of the time. But time and time again I’ve added up what the real work of the Scrum master is and it’s really no more than about 20-30% of a typical sprint length regardless of how long you’re going, by the time you take in just the things like sprint planning and updating the wall and taking the actions from the retrospectives, so what do they do the other 70% of the time? And actually what they do 70% of the time is a mixture of things, sometimes you’re coaching, sometimes you’re training, particularly if you’ve got a new team that’s just starting out and sometimes you’re just helping the team, and I think one of the things that happens is that if the team starts to get better, what I’ve found in some organizations is people then go “well, we can now get you to be a Scrum master for two teams or tree teams or four teams” and really what they end up becoming is just a team leader or a manager because they are so far removed from what’s going on which is not what the intent of the Scrum master was in the first place. So what I say to people is that, I use the ShuHaRi model, most people in an organization really need to be Agile at Shu level, they just need to know Agile, they don’t need much more than that, because they actually need to be good at something else.
Agile coaches are Ri level, people like you and I, have spent our 10,000 hours of practice really getting behind Agile and what it’s all about. But an organization doesn’t need lots of those, you only need a few senseis to go to. And then your Scrum masters are your typically Ha level, the one in the middle, which is more than just knowing the things, but they have a few more things in the tool belt, they can handle things like facilitation, they can think differently how to handle a retrospective, they can look at different ways of producing the backlog, they know that something like Kanban and Scrum exist, they may not have all the answers, but at least they know where to go and take the team moving forward. But seemingly, one of the insides of a Scrum master, if you’re not going to move to Ri level then what else are you doing, my strong suggestion is you keep your core skillset. So, when you’ve run out of all those other things to do, if you’re a developer, then pull up your chair and pair and work with a developer or just go get coffee for the QAs or whatever it might be that’s helping the team, the key is to help the team. And that’s actually written into the Scrum guide, it just doesn’t come out in flashing lights, particularly for people that are outside of our community who might be reading it. And most importantly the other part is that when we talk about the Shu level is that everybody in the team should be aspiring to Ri, but aspiring to Ri in their area of expertise, so if you’re a software developer I would expect you to be moving towards your 10,000 hours to be a good software developer or tester or project manager or, everybody in the organization should be heading for Ri, we just don’t need lo of Ri’s in Agile, in my opinion, Scrum master is that weird one because it’s somewhere in the middle and usually you have that decision do I want to continue on and be an Agile coach and be an Agile practitioner and spend my 10,000 hours or actually am I happy enough where I am and continue on doing a cross breed?
9. What’s exciting for you these days, what’s new, what’s the stuff that really has your interest?
I think the Agile in the enterprise stuff is getting a bit of attention, I am not entirely sure where I sit in all that, but I am watching that with interest because I think we’ve got the team level stuff sorted out, but I think people are still debating it, I think more importantly though is moving Agile outside of our traditional boundaries, so using what we know to try to do some sort of social good. We had a keynote here about Agile and government, even if it has to do with IT, I think there are lots of untapped areas where we really haven’t pushed it out. So, organizations, those that haven’t probably made the leap now, they’ll come when they’re good and ready, we’ve talked about crossing the chasm, I’m not sure what that means, I think Agile is well known enough, those who don’t decide to cross over will probably wither and die because they won’t be able to keep up with the pace, but I think we now need to take what we know and try to apply to other domains or work more closely with other domains to share our knowledge and as we do that we are then going to learn from them and we are going to bring those learnings back in here, so some of those things have been seen around in that area almost looking outside to bring things in, we’ve been good at that in the past by taking things like things from Dan Pink or even Lean startup, a little bit outside the idea of bringing them into the mix, that’s what’s taken my attention now, what other things can we learn from.
10. And if people are interested in seeing your podcasts, where can they get that?
Yes, the Agile Revolution podcasts that I do with my colleagues, Rene Troughton and Tony Ponton, out of Australia, that’s theagilerevolution.com and if they want to know more about me they can find me at craigsmith.id.au.
Todd: Alright. Well, thank you very much for doing this, Craig.
Thank you.