Sure. Well, I started off as a programmer in various contract positions in university and that lasted for about four- five years until the first recession hit in the early ‘90s. Then I did a stint in a tech support department for about eight months and that’s an eye opener when you go from heads down changing code to dealing with customer problems and not being able to change code and the next summer I had a position in QA, which was a nice happy medium, suddenly I remembered all those problems that customers were reporting and I could influence dev to actually change it and help raise the quality. And I was bitten by the bug and I have been doing that for over 18 years since. I guess I reached a point at which I was learning much less in the day to day life from the people around me and so I started teaching and presenting at various testing conferences and about four or five years ago I met someone who really caught the gist of what I was talking about in one of my testing presentations at a local university and said “you need to come to an Agile coach camp and talk about testing” and I was like “what is this Agile thing that you talk about?”. And I did and as soon as I started hearing some of the other Agile coaches I was like “you’re talking about quality, you’re talking about values, I get this stuff”, but none of them were talking about the testing experiences and skills that I was, so I really saw the complimentary fit there, so for the last four or five years I’ve been participating more in the Agile community and it was last fall that I became aware of this Agile Alliance conference and I thought “well, I’ll submit some topics” and I got accepted and it was kind of cool.
Sure. So the first one was called Pitfalls in Agile Testing and How to Avoid Them, I’d done a similar presentation locally in Toronto and I just used a big giant whiteboard and we just started with the words Agile testing and we just blew it up from there and it was really nice and interactive kind of discussion and I turned that into more of a workshop where I figured there will be more than ten to 15 people showing up, actually there were about 100 people, so what I did was I took some of those main ideas and I turned them into discussion points that we did like a world café style, so at each table was one of the topics, and as people joined the table they were just joining the discussion, and they would take notes and doodle and capture their thoughts and ideas and each of those sessions last for about 15 to 20 minutes and then they move to another topic, they move to another table, so the tables or the ideas are always being refreshed with new people and new ideas. And then after about three of those sessions return to the room and we started to debrief, what were some of the main points that you got from them. It was kind of cool because I got some really great feedback from them afterwards from certain individuals, where one person in particular said that he stayed at the same table for all the three sessions on automation because he was most interested in that topic and he said it was really interesting that every time a new group of people joined he was learning more and more from the people participating in the discussion. And I told him that was basically the point of that format, was to allow each person to get a chance to share their ideas, what they’ve learned, what they’ve tried rather than listening to just me lecture at them, they get one perspective, they have the opportunity to have like 30, 50, 60 perspectives on it and he said he found that incredibly valuable. And that was cool. And that was just the one. There was another aha moment that some people had that indicated, when I talk about types of testing and a lot of people tend to focus on heads down, sort of very narrow focused kinds of tests which are actually more like checks. And so I clarified the distinction between checking and testing.
3. Maybe clarify that for us actually, so what is the difference between checking and testing?
When you write down a set of steps and it’s very narrow in focus and then you have a very clear set of expected results, then what you are doing is you’re putting these blinders on to everything that is going on around you and you don’t see any of the problems that are going on. And in psychology this effect is called inattentional blindness and it’s the inability to see something that is right in front of you because you are just so focused on a particular task. And if you go to a website like the invisiblegorilla.com you’ll see some videos that are based on the original psych studies and they have these moonwalking bears going by, and even when you’re expecting the moonwalking bear you still miss other things that are going on around you and it’s really kind of cool. And we realized that as humans we are imperfect beings, we are always going to miss something. And so certain kinds of tests, whether done manually or even if we have a computer run a test, like a unit test, acceptance test, if it’s scripted the computer is blinded to only what it was programed to look at, that would be what we would call a check. It’s not really looking around, it’s not discovering new information, we need creativity, imagination and those kinds of investigative activities what we refer to more as testing. So we make that distinction. So when I clarify this people are like “oh, most of what we do is checking, how can we add more testing and investigation to what we do?” And that was some cool feedback to hear, too, and I’ve heard that every time I’ve done this particular workshop and that’s one of the insights that they gain into the robustness of their testing activities.
There’s a lot of definitions for that; I’ve heard some good ones here, too. Jez Humble, the author of Continuous Delivery, in his presentations indicates that there are two rolls he expects a tester to fill. One is to act as the customer or user representative and two is to raise the visibility of the quality, the transparency, make the system quality transparent. And I’ve heard that worded in many different ways, that’s just the one I’ve heard today, twice actually. I’ve heard more technical definitions like it is a technical investigation geared to uncover quality related information for stakeholders that matter, that one came from a lawyer and a professor, there’s a number of different ones. Generally it’s an exploration, it’s an idea of taking something that’s important to someone and looking at the product or the system in a more creative and structured ways to uncover information that’s important.
Well, every time I run that particular workshop I always learn from the people that are there. Another nice thing that happens is when I pick a topic I always tend to struggle with what’s important to the audience, I don’t know, so an activity where I can get them to tell me what questions are on their mind or what issues are important, then when we get discussing it and I can see where the energy is, I can see if I have stories or experiences that can help answer or address some of their concerns, or if there are tips or resources that I can point them to. And every session is unique, people’s problems are always unique. There’s a second part to this particular workshop, the first five pitfalls I can say “I can explain this to you and here are some of the common traps”, it’s really obvious and they discuss around those, but sort of a bonus pitfall, I have seen only once or twice before in the last 20 years almost, and that’s what happens when the testing gets too good, and then you have to start asking “what happens when the testers start getting ahead of the devs and the development?” And so I constructed this simple exercise to help them illustrate that. The reason why I used an interactive activity was because you can’t explain this, you have to experience it, so I had them create something and then do a test strategy, just using personas or something, simple devices, then execute those personas against what they had created five minutes before and then just share their insights.
So, yes, it was intentionally set up as a little mini waterfall, it took ten minutes to do all of this and then share their insights. And it was really great seeing them “oh, the testing wasn’t fair”, “why isn’t the testing fair?”, “because they were testing things we haven’t designed”, “well, have you ever seen this happen in your development projects where testers are reporting bugs that weren’t considered during the development phase, why is that? Are they testing things that aren’t important? Ok, that’s easy to address, or are they testing things that are important but you didn’t know about, why didn’t you know about that?” So, I’m trying to uncover elements of the Agile process that haven’t quite gotten to yet, which is the importance of collaboration and the integration of ideas from different sources to develop not only collaborative good software that provides value, but no surprises because no one wants to be surprised afterwards saying “oh, you’ve got all these problems because of all these things you’ve missed”, now they have to be added to the backlog later, maybe well get to them maybe we won’t, but the dev might feel bad that their code isn’t at the high quality that they hoped it would be because they’re getting that feedback too late. So it’s a nice little simulation to help them understand that when you get to the point that you’ve developed all these nice models and testing skills, remember to collaborate, remember to include them in that process so that they’re not left out because that wouldn’t be very Agile.
Sure. Well, the first one I talk about is there is no Agile testing and this is a matter of semantics but I don’t use Agile as an adjective, I use Agile as a matter of context and so in that context everyone does testing, however we don’t have Agile code so we don’t have Agile testing. What we do is we do testing activities in context and so we don’t have Agile testers, but we have developers that are working in an Agile context. If you want to use the Scrum definition, there’s three roles that they recognize, there’s the product owner, the Scrum master and the developer. And the Scrum guide is clear about saying “we only acknowledge the title or the role of developer”, so if you’re a coder, a tester, a designer, a tech writer, you’re developing something towards the value that is being delivered and it’s really just meant to reinforce the idea “let’s not create labels that separate the teams, but let’s try to focus on working together and trying to deliver this one common goal”. The second one was “you're not Agile”, well you’re not going to get far with Agile testing if you’re not Agile might it be at the individual, the organization or the team level that really needs to be congruent across all levels. The third one was “you’re not doing testing”, that’s where I started getting into checking versus testing, are you relying on manual testing for regression testing which is you’re not going to be Agile for very long if that’s a primary mechanism for doing that, so a little discussion around what is testing, what are some of the various elements that you need to keep in mind. Automation is often a huge pitfall, it’s like “well, if we need to get rid of manual regression testing then we must look to automation”. I hear it all the time, people are looking for that one automation solution that is going to solve all their problems, well it’s not like that, let’s focus on where’s the value to automate certain tests and let’s find the right tools to do those jobs, let’s not necessarily put a lot, too much effort into all the frameworks, let’s build it organically based on what we need to accomplish at the time and I’ve seen some organizations go heavily into very expensive automation frameworks that just wasn’t providing that kind of value. So let’s be aware how far we go down that particular rabbit hole.
Yes, well that is one of the questions that I am asked most often. I often start by showing Mike Cohn's automation pyramid. We want to build a nice stable base let’s start with the kinds of test that provide us confidence in the reliability of the builds that we’re producing. So we need to start at the lowest level. If the devs, the programmers aren’t doing unit testing or they’re doing it manually then they’re wasting their time because they’re paid to automate, they’re paid to script, so they may as well script the tests that they do as well. But there’s a lot of low level kinds of structural things in terms of is the bill process automated, or APIs, let’s not test those manually, we should have those, what about interactions with the data, the databases, security, permissions, there’s a lot of sets and components and aspects of the system under test that we can automate quite nicely, and we should be automating it, so that as we continue to build it, we know that we have confidence, it becomes our safety net. As we move more to integration layers and aspects of the system the complexity goes up, then we get to the top layer, the UI or the end to end kinds of tests, this is where your traditional QA and testers come in, where all the big product venders come in, and those tests using traditional methods tend to have very low value because the tools are set up for that inattentional blindness that I told you, you hard code very specific paths through the system and it doesn’t look around, so be very attentive to the kinds of tests you want to automate, those end to end tests.
And those you really want to focus on the flow, again using Jez Humble’s terminology, what’s the user’s journey through the system, what are the things that are important and let’s focus on those scenarios and let’s keep that to a manageable level, usually if you can relate that back to a persona that matters. There’s a set number of personas for any given system, we can focus on what’s important, we’re not trying to automate all tests, testing is an infinite activity. So we want to know that the automated tests we do are providing value, that they’re stable, that they’re reliable and these are hard things to do, making sure that they’re not providing false positives, false negatives and when it goes red that there’s a problem somewhere in the system that we know we need to investigate. So you need to start with that strategy in mind and then work with the team and the technology and find what works for you in your given situation, is there a prescriptive model or tool set that I can say in the last 20 years every team that I’ve worked with has had a completely different model implementation. I know that at the UI level, when I am doing exploratory testing, I like to use Ruby or Python because they are very simple and powerful scripting languages, they’re great for doing throw away scripts it’s like “oh, I need to collect some data, I can write a script in 15- 20 minutes, it gives me good information and then I don’t care about it anymore”. So automation is a powerful tool, build it correctly, it can be a huge rabbit hole if you invest too much time without proper guidance and structure.
Good question. I often write those two words down on the whiteboard, I say the first part is test and the second part is automation, you need a tester for the first part and you need a programmer for the second part. And I’ll say that with a caveat that if you want good reliable automation then you need to treat that code just as reliably as your production code, it needs to be reviewed, it needs to be something that is providing value, it needs to be maintained, it needs to be version controlled and programmers have those skills, the average tester, the high level system tester doesn’t, you have people with varying degrees of skill set in between, but I’ll talk about those two sort of more extremes or ends. Paring someone who understands the testing domain with someone who understands the programming domain is a very powerful combination and another benefit of that is the camaraderie that is built up in the team that a tester knows that if they have issues with this is not their problem to solve alone, but that it becomes the whole teams problem to solve, and they don’t feel like they’re inadequate because they don’t have programming skills that match the programmers’, and the programmers don’t have to feel bad because they don’t really get that testing stuff but they can rely on the expertise of a tester to do that. Having said that, when I do exploratory testing, I usually use a different term than test automation, I say computer assisted testing or computer assisted test design, and so when I am investigating the system in a particular way for a particular purpose, if I feel I have something that a computer can help me do I will look for a way to try and get the computer to help me get that information.
For example, when I was testing a web application, there was a financial app and I had this new report and it had a link to the formula for all the numbers on a particular page and I was just standing back and I was like “there’s three pages of links here” and I’m like “how do I check them all”, this is 1.0, someone should check all these links, I’m not doing that by hand, so I called up a little command prompt in Windows, I typed in irb, I loaded the watir gem library and I connected to the web browser and I said “how many links are on this page?” and said “168”, and I’m like “glad I didn’t count manually”. And then ok, let’s write a script in Ruby and Watir to drop that into an array, put some identifiers, and then click on everyone and if it gets to a page, then just hit back and go on to the next one. I did actually find a bug, the script found the bug for me. Now, I guided the test, I knew exactly what I was looking for and I programed the computer to help me do that task because otherwise I would click- check- back, click- check- back, and then “did I check that one?”, but a computer can do that quite quickly. So in that sense, do I call it test automation? No. But the computer helped me to do that and I used scripting as the tool to do that. So, yes, I do encourage testers doing UI or end to end or exploratory testings style to learn about scripting because it can be a very powerful tool to help them test more and efficiently.
Yes, they are. So when I work with teams now, I think that’s an interesting transition that I’ve made since joining or becoming part of the Agile community is looking more at the whole picture. I used to be focused on just the testers and the QA community and then realizing I’d hit a wall in terms of I can’t influence more quality by just focusing on this subset of the team, I work with the whole team. And before I get into any discussions of test, techniques, I started very high level with what is quality, do we all understand what quality is, are we on the same page? And that’s always an interesting discussion because everyone has their own ideas and that’s kind of the point of that exercise, actually it’s a workshop that I do with teams, it gets everyone on the team talking about so that they’re all on the same page. One of the things that always comes up, especially in the Scrum framework is “what is done?”, how do you know what done is if you don’t know what quality is. And so when you start talking about quality and the fact that, you know Jerry Weinberg’s definition is quality is value to some persons at some time, because they may change their mind about what quality is later. But some person, wait a minute who is this person, that means if I ask different people they may have a different interpretation about what quality is. Yes, and this happens.
Nice thing about personas is that it tends to group certain definitions of quality together, in terms of what people care about, and so if we start to identify personas with that definition of quality we can see how we can start to come together with some ideas of who we think people are that are interested in our deliverables and what are important to them? That starts to form the basis of testing strategies. And that can help guide exploratory testers, system testers or developers. Then I can engage the devs with ideas like inattentional blindness, “ok, if you’re looking at this alone, what kinds of testing or interactions are you doing with the system, how are you checking to make sure that you’re not fooled or blinded?” Same thing with building code, the whole idea of paired programming is that two heads are better than one, well if you’re testing alone you may miss things as well. So there’s techniques that you can do, whether you have resource monitors running in the background, checking for CPU spikes and memory leaks, and disk usage, network traffic, that can be your second pair of eyes, having someone that’s pairing with you, like a dev and a tester pairing together, going through some simple heuristics or models. There’s easy steps that you can do to introduce them in an Agile fashion that isn’t painful, and I found out this tends to work well for me when I introduce it in this fashion, sort of “are we on the same page of what quality is?”, how do we think we want to tackle that in terms of the testing activities that we do. It becomes very flexible, it’s sort of a high level model or framework that allows them the flexibility to choose what tools or methods they want and it’s an easier transition rather than saying “oh, you need to learn this this this”. Disengage! So I am much more subtle in terms of helping people become better testers that way.
10. So, are there any other talks that you cover or you want to mention, lost that train of thought?
Well, actually, last one was with regards of types of testing, is that you often hear “oh, we need unit testing and integration and system testing” Really? What do those things mean? You can get stuck on jargon, and where the jargon fits in with your processes and I often show Brian Marick's Agile Testing Matrix or in the book Agile Testing by Lisa Crispin and Janet Gregory they call it the Agile Testing Quadrants and when you look at these four quadrants it really helps form part of your complete breakfast, it’s all in there somewhere. You have the testing activities that support development, you have the testing activities that critique the product, the ones that are technology facing and the ones that are business facing. So the kinds of users that are business facing and that critique the product are users, business analysts, they are the ones that will tell you “did you solve the problem?” More the unit tests would be the ones that the programmers own and that’s down in quadrant one because it supports the technology as well as the development effort and that really is your safety net. And it’s really neat to see where everything fits in and it’s not that you have to do something in every quadrant for every release, but think about the kinds of tests that provide value to your release now. And that can be something that people can get hung up on, what kinds of testing are we doing, let’s not worry about labels, let’s worry about what provides value for our stakeholders or our customers now. It goes back to the discussion of the quality and who were the people releasing for right now, do we know what they care about then how can we test to find out that it meets their needs to a certain degree.
Todd: So, your other session, kind of had a bit of a nice draw on the title, it’s called the Zero Page Test Plan.
Yes.
Todd: So, maybe give a review of what that was about.
Sure. So I’d say about seven or eight years ago, when I was getting more into exploratory testing I had already been doing it for a few years, I started to see certain patterns in the test notes that I would keep. So these exploratory sessions you basically would have a mission, you go, you look at the system from a certain perspective, try to gather certain kinds of information and you don’t really have any other requirements other than gather this information, identify what the risks are, try to mitigate them. You find problems along the way, have lots of conversations and if you want to learn from that, if you want to collaborate with other team members, then you have to keep some sort of a record, whether it’s a mind map or a set of notes, there’s lot of different tools out there to help you keep track of this. I was looking over my notes and I was finding certain patterns in terms of “on this day when I was testing this feature, I was in this environment, testing with this browser I was covering off this particular risk or quality attribute” and I started to see a pattern over time, there were always three different things that I was looking at in terms of the environment, the quality attribute or major risk and the product area.
And I started to play with the idea “do I always need to put those three things or can I take certain things out?” and as I would remove something I would find looking at the notes later “why was I in here again?”, so I just drew this little three dimensional model, you know my background was in physics so I love free body diagrams, I doodle all the time to help explain testing concepts and people like my little drawings, so I just drew this orthogonal graph with three dimensions and “oh, maybe the product features under test are on one axis and the environments or the configurations that I am testing are on another and really the quality attributes are on a third”. And I really started to see “yes, that’s a patterns that’s starting to emerge from my testing notes” and I would think about it little bit more, is there something more to it and I realized that was also tied to, because you could look at the functionality of something we care about, I’m testing the functionality of this particular feature in this particular browser or on this operating system, and you get this idea where we fixed two, then we can play with the third or if we're changing all three we are constantly looking at new areas.
But I realized even if I fixed all three, if the build changes that’s another dimension of the testing strategy because problems can occur when you are doing regression testing, where, yes, I am doing the same functional test in the same environment for the same feature but as the build changes bugs may appear and disappear, so the builds themselves is another orthogonal point and another aspect of it is even time, so we can have a fixed quality attribute that we’re investigating in an environment on a build and in a particular place and even with the same build, over time, we can start to see different problems emerge and we would have to reinvestigate that “we tested that yesterday, it’s the same build why doesn’t it work today?” oh, we deal with the investigation, we discover that someone had changed the configuration in the back now this is working differently or there’s a memory leak here and this has become so slow that the performance is horrible, so we can see aspects of the quality attributes that are now degrading over time so time became a fifth dimension of this overall testing strategy. I basically extracted this model empirically looking at my testing notes over a period of several years and then I started to present that model back to new testers and I found there is a very quick way to engage them on the different aspects of their testing strategy really quickly.
So the Zero Page Agile Test Plan is just that, I present the model as a way to have a device to collaborate with the team members on what do we feel are important right now for this release, what are the features under test or stories, who are the personas or the quality attributes or what environments do we need to test and this is really good for manual testing, exploratory testing where we’d have to cover off a lot of different aspects quickly, but then when we learn about that and we want to automate that we then know precisely what information we are gathering and when. A traditional test plan from the more waterfall days is this huge monolithic document, I think it starts officially at 20 pages and I’m not sure there’s an upper limit, I’ve seen ones as big as 80 pages and no one reads them and it’s this artifact that test teams are told they have to do this, they need to do this, no one can tell us why and developers don’t have to produce a development plan so I don’t understand why we have to produce it a test plan, but when you look at the various elements in the structure of this document, it’s a project plan.
And so when you are looking at an Agile team and testers trying to join, it’s like you don’t have to create a project plan to begin your testing, in fact that is the wrong tool to use, so this Zero Page Agile Test Plan is like recognizing that when you’re becoming more Agile you start to ditch some of these things you’re told you are supposed to do but really no one uses it and it doesn’t provide value. And then what are the devices that we need to try to do to increase collaboration and discussion around what some of the important items are that we need to get information about right now. So that was really the point of that discussion and so I actually had them build real models and play with it and put little stickies with the various things. It’s funny, people ask me “are the slides going to become available?” I’m like there’s only one slide with the model and you built them, so you get it now. And it’s not that they have to walk around with this three dimensional model but they can now go to any discussion and when people start talking about features under test, they can start asking questions automatically about the various elements and how they work together. So it’s an useful tool for testing anyone on the team, I have project managers, sorry, product owners, Scrum masters, developers and testers in my sessions so there’s a wonderful mix, I said this is a team tool to understand the testing strategy, it’s not a tester tool.
11. So, is there somewhere people can go to learn more about that model?
You know, I’ve been asked about that a few times, I haven’t published it, not yet. So I am making the slides, the slide available on the Agile 2012 website, I think no is a short answer. I will be publishing more information about how I’ve been doing testing later and I haven’t decided on the format for that. Over the last several years as I present some of these little tools and devices, people keep asking me “oh, you should write a book”, maybe, and I think that might be something that I might just collect those ideas together in a place that tries to break it down as a useful guide. I’m a big fan of handbooks, not of actual textbooks, so if I do write something it will have to be a “choose your own adventure” kind of story where you can jump to any part and say “oh, this is the problem that I have today”. So that’s an idea that I have, I need to set aside the time to start writing that though.
Todd: Ok, well, hopefully we’ll hear more about it soon. Thanks a lot, Paul.
Thank you.