1. Hi Ron! Could you tell us a little about what you're doing these days?
I've been hanging about in the Agile movement since pretty much the beginning, trying to learn what it's about, trying to help other people learn what it's about - and I hope to continue doing that until everyone else dies and I'm left here in the world all alone.
I think the thing I'm trying to figure out is: what the essence of Agile Software Development is, when we look at it from the outside, and what that causes it to become, what that causes us to need on the inside. If you think about what Scrum says, if you think about what Alistair's Crystal Clear says, they are all about: get together and build software that really works every week, month, whatever period of time, and everything will be ok. I don't happen to believe everything will just automatically be ok, but I do believe that the process of going in a circle, building software that really works to the specification of your customer, is the essence of what makes things happen. So what I'm trying to do at the moment is to write down what that means in a way that can initially be read by a would-be manager, product owner, or customer for a product. I want to say: "Here is what you must do in order to make your team satisfy you." And then within that circle drill further and further down to how that customer/product owner does things and to how that development team does things, /so as to be able/ to deliver the running, tested features. So it begins and ends with the notion of "Let's ship software, to their spec, that really works."
Not really. Extreme Programming begins with a planning cycle that is similar in structure to the one in Scrum, though it's actually derived from a different source, that has to do with taking a long-range view of what you're doing, but then breaking it down into iterations and at the beginning of every iteration, figure out what you're going to do (which is analogous to the Scrum "backlog") and then doing it. So XP begins with an outer planning cycle that's essentially the same as the one in Clear, as the one in Scrum, as essentially any of the the iterative planning cycles. XP is rather unique in that it goes on and describes specific technical practices that might be valuable, and that, I think, has some merit. What I'm interested in, in this work I'm doing right now, is to drive the need for those practices - because the real practices we do in software development are far more rich than any dozen you might name, and so the idea is to have people focus on what they're trying to accomplish, and what is implied by the need to ship this software, every short period of time. So I kind of begin with that picture, figure out things that are implied by that, and end with it, and I'm trying to put all o that into a book that can be on sale in airports, and everyone in the world who wants software will buy one!
Sure, I'd love to. I think of software development projects as always starting out the same way: and I draw a little picture that looks like this. Here's time going along on your project, and here are the features that you're going to need. "They" (the people who tell you what you have to do) always know a whole bunch of features that you must absolutely have, that's how projects start. Even more importantly, they know a time, a date, at which this project must be done. Every project starts with "I need all of this stuff, by this absolute date" - that's the target. Now, in Agile right now, people are coming to a partial learning, I think, which is that you can't really know how much you can get done by any given date. And so, the current state of the community, as I read it, is we don't want to tell anybody what they can have by when. The problem is: this is in fact the valid way to define a product, "We need the product by Christmas, and we need all this stuff done." So to me, Agile software development is about looking at that chart and figuring out what can really be done. Now there are two ways that software is historically done: the old, waterfall-ish way and the Agile way. Let's look first at the waterfall way. Waterfallish projects do a lot of requirements development, and designing, analyzing, coding and finally they're going to do a little testing, and stuff gets done - build a lot of infrastructure. So when you think about what you can learn, what you can see, in a waterfall project, there's all this activity but it doesn't mean features. So the project goes on and on and on and on, and the plan is that sometime, way down towards the end, you're going to suddenly start doing those features for your boss, and you're going to zip right up and hit the target. That is the typical software plan. Brian Marick, in a different context, likes to say "that trick never works," and it never does. What happens instead is that we start later than that, and we produce software at a much slower rate. This leads to what we call "unhappiness" in the corporate levels, and this runs downhill and makes us unhappy as programmers.
Well, it's not my idea, it's the Agile idea. I'm not sure I've invented any of this stuff, I just talk about it a lot, but in any case the Agile idea is that instead of building this invisible stuff, you would build features that actually make sense to your management, to your customer and so on. The Agile idea is this: we will build features at a rate that is more or less level but it goes up, so that you can see what you want - and every so often you a release of features that you recognize as part of your product. The idea is that we will be able to project that line and see what's really going to happen. It's quite possible that you get the same line. I think Agile projects can go faster, but my point here isn't that they go faster it is that they are predictable. The thought is: if a manager can look at this chart and say "Oh it looks like you're not going to deliver all my stuff by my date", he has options that he doesn't have if he doesn't get the word until the last possible minute. His options include: "Would it help if I added more resources, could you go faster? Would it help if you were smarter (are there ways I can help you get smarter?), Would it help if I slipped the date?" I think the most important, at least in this country and in many countries, is the date, because you've got marketing campaigns, or it's Christmas etc. Or "Can I figure out a way to ship this many features on that date and make it into a viable product?" To me the fundamental value of Agile is that it produces information that managers can use to make better decisions than to just hit the programmers with sticks and hold their feet to the fire. Ward Cunningham once said that he doesn't think that programmers program any faster because they have hot feet.
It sure is. I think in essence that's the point that I take from Agile software development. Instead of being able to say: "I need more and I need it sooner," this puts management in a position of realizing that what they have, in essence, is this great big blob of "project idea" that would be the perfect product if you could get it by the time you could get it, and that you've got this little bitty box, which is the capability of the team to perform, and you're trying to squeeze that much product value into that little box. What it does is it changes the customer's focus from "You must give me more software," to " I must choose the features and the sub-features that make the best possible amount of this stuff go into the box." There are a lot of junk features in everybody's product ideas and there are a lot of really essential ones. You want to push the essentials in and leave the less essential ones for later.
That's true. It is also not the essence of what I'm talking about. It's kind of the outer control of what I'm talking about. I think the project is controlled by the process but it is not executed by this process. If you want to do this, and I believe you should because of the controls you're talking about, there are implications about the way we work, that can be derived from this picture.
Let's take a look at the picture again and see what has to happen in order for this to go on. We've got this timeline and this list of features, and we're trying to build features in a relatively constant order so that we can project that line out and see when it will be done, relative to when we want to be done. So we've got some target out here: How are we dong against it? There are thing that can make this not work. For example, we claim to be shipping features: if we're going to hit the target and be ready to ship, those features have to actually work. To the extent that those features have bugs in them, that amounts to a negative feature. You think you have ten features and there are two bugs, you have eight features. So a bug is a negative feature. If behind the scenes there are these negative features being created, this line that we think projects like this, doesn't. It's somewhat lower, but since we don't know how many bugs there are, we don't even know how much lower. So that somewhere down here is this fuzzy reality, below this really sweet-looking line. So for example, therefore, a team that is trying to deliver running,tested features: they really have to work. Which is why I use the phrase tested features, to mean that we're certain these things are working. One impact on the team is that the bugs have to be eliminated as they go, they cannot be eliminated in a test phase at the end.
There are a lot of things to it and it's worth emphasizing that I think software development is made out of hundreds and hundreds of skills and not the three or four that came up here. But there's one that is next on my mind, when we look at this same chart. Here's my features and time again. Here we are now, with features that are working, that are fully tested, predicting some future date where we ship a wonderful product. To the extent that you're going to be able to do this, you must start with kind of very simple system. If I'm going to ship a feature in the very first week, I can't possibly have done a huge design and infrastructure that we normally think goes into making software be good. So we know that at this point on this chart, there's only a little tiny bit of design in there, a little tiny bit of infrastructure, just enough to support this feature. If we did not do that, if we built infrastructure and did a lot of design upfront, then we would get that other curve that we had, that is invisible and then suddenly is intended to leap up (which it never does). So, we're working with a simple design and moving forward. If we do that, the amount of design that would support this much software could be very simple. But as we get out here, the amount of design you would need to support that much software... you would need more. So there must be more design going into the software every week as you go forward, till the very end of the project. If you do not do that, again you get bad predictability, because we want to say: "Let's produce these features and in a constant order so we can see where we are going." If we fall behind on our design, what happens is: our velocity, instead of going up, will begin to tail off and ultimately crash. Almost every software developer at one time or another has worked on a program where he wants to say : "I can't put another feature into this; this program's design is so horrible that I can't add any more features to it, it's incredibly difficult." Our natural inclination is to say: they should have more design on this thing before they started. The truth is they should have done more design on it every day, because every time we make a change to a program we are detracting from the design, and if we're not to ultimately detract until this curve happens we need to put more good design back in, to do what is called refactoring (after Martin Fowler's wonderful book, Refactoring). So, this curve says to us that we must design all the time, and we must develop the skills to design all the time. So now we have two elements that programmers have to know in order to do this thing: they have to know how to produce code that actually works, and to prove that it works, in my opinion, but they also have to know how to produce software in this incremental fashion by adding design quality to it all the time and by keeping its design quality high. What happens on the picture is never that you just track needs along your design. What you may do is a little too much design at the beginning, which will actually slow you down, then you start doing some features, then you start not doing enough design and producing more features but that starts slowing you down, and so what you try to do is just enough design. R-ecognizing that some days you'll have a little too little, some days you'll have a little too much. You're trying to find a balance of features production with design insertion that enables your feature production to stay at a constant speed. Too much design, you slow down - too little design, you slow down.
I think a project can unfold that way. The question becomes how long can it unfold that way, and what does it take to do it? If we are growing these features that way: putting the design in as we go, how long will this product last before you get that inevitable fall-off that we all recognize as "this program has become so crufty that no one can touch it." It's very, very bad to have to rewrite a program, it's incredibly costly, and that's another trick that almost never works. Whenever a team gets involved in having to throw away an existing system, and essentially reproduce it, they wind up in a chase situation they can never really, quite, resolve. Because there are features of that program that are important, that people can't live without, and therefore a this new team will have to produce essentially everything in that product. Usually that product is still being maintained, which means that at about the time the team has put in all the features that the product used to contain, it has got new ones, and so they're always chasing after it to try to catch it, nipping at the heels of this slowly advancing behemoth of a project. I don't think anyone knows how far it is possible to push this process. What we do know is that what causes this drop-off is that the design of the product as it exists no longer sustains the capability that it has. And that says that whether you design a whole lot up front, or whether you design it everyday (which makes more sense because why would the design from ten years ago be useful to this product today), no matter which one you do, there may come a day when the design is no longer good. I likely to come about because the design has gone in some horrendously wrong direction for years and we suddenly discover "Oh I wish we hadn't designed it like this at all?". I don't think that's likely, because I believe that good design has modularity. We look at a program and if it's one of these "big blog" programs, it's not really good. But if we divide the program into nice, neat little boxes, then even if we don't like what is in one of these little boxes, we can change that box and because it's interfaced correctly with the rest of the system, we don't have to rewrite the whole system. So, I believe that the old ideas of modularity that came about in the 60s and 70s are still valid. They say "build your system of small cohesive chunks hooked together with simple interfaces and you can live forever." In that situation it is possible to take a look in any given part of the system and say "now we don't want this to work in a hard database connection over some kind of amazing phone line, we just want this to be a web service." Well, as long as this guy and this one are just talking over some connection we can just change the meaning of that connection. If we don't like the way this particular database works we can pop it out and pop in another one. It ought to be possible to go forward for a very long time until what has changed is not the design of the system but the concepts behind it. Sooner or later somebody is going to look at a giant product like Microsoft Word and say : "You know, that's really not the way we wish we could do word processing. It shouldn't be like that, no matter how it's implemented. It should be some other way." And when we think ot that other way , that one we may have to start over. We may be able to use some of the components of the other one, I would think paragraph formatting would be paragraph formatting, however you did it, but I'd like to move in a direction that says: a radical change in product direction is brought about because the product owners have a better idea, and not because the software owners had a bad idea.
The good thing about this, I think, is that there are hundreds and thousands of interesting exciting skills that you have to have to do this work effectively. It isn't enough to be able to code and design and test; you have to be able to interact with your customer, to be able to help them analyze things, to help them define their requirements in terms that are able to be tested concretely. You have to then be able to sit down and work together with other programmers to build these components and put them all together. There are in-depth technical skills. It's easy to say: you have to test your code. There are yahoo groups, testing courses all over the world, there are acceptance testing tools like Fit, there are programmer testing tools like JUnit and NUnit. A good team that does this well has all that stuff under its control, which means that anybody on an Agile project has the ability, and essentially needs, to grow in the depth of their technical knowledge but also in the breadth, because if I can only do, for example, the database part of the application, then I become a kind of a road-block in that project. Either there will be database work piling up, waiting for me to do it; or I will produce a whole bunch of database work that the project isn't ready for. So, I can't let myself become too narrow, because the team doesn't always need my specialty, sometimes it needs this specialty or that specialty. The more able a developer is to spread over the entire spectrum of the project, the better they are going to be able to do. So when we start looking at all of the things that are talked about at the conferences, all of the different mailing lists and so on, they are all important. The trick is to figure out what you want your place to be, in that. One of the mantras on my website lately is "how good do you want to be, and when?" It's a choice: now, I don't do everything I ought to do, or even everything that I'm quite confident that I have to do, and I don't think anyone does. But the more skill we could bring to this project, and the more we learn how to put those skills togetherwith other people's skills, I think that enables us to come around to do what we have to do.
I certainly can try to that. The point is we have a target; we want to produce software at a rate that tells us how we are doing against that target. It's realistic to assume that they're not going to get everything they have asked for, because their job is to ask for more than can possibly exist. When you envision a product, you envision it sooner than it can possibly exist and better than it can possibly exist. If you can complete this line of software in a nice straight growing increase of visible software units, that will induce in management the desire, and provide them the capability, to feed us the features we really need in the order that we need them, so that the value of this product grows constantly and so that ultimately it is shippable at any point in time, and I think that's going to bring about a better cooperation between the people who envision a product and the people who develop it.