1. If I could ask, what have you been doing lately, in the last few years?
I fell into a line of work that I never really anticipated when I got a phone call one day from a lawyer who asked if I would be willing to work on a lawsuit involving $150M project failure, which apparently goes on quite a lot - at least in this country not around the world; the rest of the world is not quite as litigious - but we tend to have lawsuits whenever there are big problems, to figure out who should be blamed and $100M software failures are fairly common these days; so I’ve gotten into that more and more in the last 10 or 15 years.
2. And through doing that, what can you tell us about the industry?
Well, of course, it might be a little bit misleading because I’m seeing the situations that have gone so bad and have led to such disagreement between the parties that they ended up in a courtroom and you really don’t want to generalize too much; but the discouraging thing is that these project failures, which we continue to see even today in 2012, are really management 101 failures. They’re not caused by some very esoteric brand new technology that somehow mysteriously failed; they’re caused by lack of requirements or misunderstood requirements or requirements that changed so often the vendor couldn’t keep up with it or lack of participation by the customer. I mean anybody who’s worked in the industry for five or ten years could probably write down a standard list of ten major reasons a project is likely to fail and those ultimately turn out to be the causes.
Of course, the interesting thing is that it’s never entirely black or white; you never have the absolutely guilty vendor or the absolutely guilty customer for that matter, it’s more likely to be 60/40; but the question is when did the problems begin and who should have been aware of them and were those problems hidden or shared, etc., etc., etc. and unfortunately those continue even today.
3. And these would tend to be outsourcing type projects, I’m guessing?
Primary so yes; I mean if only for the simple and obvious reason that a company is not going to sue itself if some internal project team runs into a problem; of course if the internal project team screws up on a product that is used by the external public, that can lead to a lawsuit; but more often what I’m seeing are lawsuits between a customer and some external vendor. Now it may be a package vendor, the classic being ERP systems, but it also happens quite often these days with the very large system integrators who promise to come in and develop and bring in other external systems and put it all together and deploy it and so forth; that’s where most of the trouble seems to emanate.
Well it’s the latter; the vendors of course have all heard of Agile and most of them would certainly agree about the likely benefits and they often put it into their proposal; but in most cases what they’re doing is just using a popular buzzword that is interpreted as “this is magic that will let us get the job done faster than would otherwise be the case”; now the practicality of actually making that happen often gets lost; the idea of the customer and the developer being co-located so they can have continuous interactions with each other, well that doesn’t go over very well if the developers are on the other side of the world working in a software factory with one representative at the customer location desperately emailing stuff back.
So, Agile is being proposed and promised and maybe some bits and pieces of Agile are being carried out; so maybe within the software factory on the other side of the world they’re trying to do continuous integration and responding to incremental changes and so forth maybe some automated testing; a lot of the bits and pieces that Agile people would expect to be part of a larger equation; but they don’t usually go all the way through the process the way you would expect and the way you often see with an internal project team.
Yes; it was a book that I published about a year ago now that essentially consisted of interviews with the CIOs, chief information officers, a couple of those of the largest American and British firms so Google, Microsoft, American Airlines, a couple of the big telecommunication companies; the CIO of the United States Vivek Kundra loved Agile; he said “now it will let us get projects done within three years” and I said “what?” and he said “well it used to be ten years”.
So the questions that I had were pretty common: I asked them what you were most terrified of and they all said security and I also asked them on other things, what do you think of Agile and every single one of them to my great pleasure thought Agile was wonderful; and when I said “what does it mean to you?”, it means “faster delivery, somehow”; somehow the development teams are doing something that lets them get stuff done faster; that was their understanding and it didn’t really go much deeper than that. So the idea of them embracing change or delivering something the customer actually wants and uses or delivering higher quality systems, they didn’t talk about that very much; and they wouldn’t have cared, I think, very much about some of the technical enablers, the continuous integration and testing and so on and most importantly, it never dawned on them that to really encourage and support an Agile culture and mindset they themselves might have to change; that was lost on them. So it’s encouraging that they think Agile is good, but it was a little frustrating to see that their understanding was fairly shallow and superficial for the most part.
Well I think it is a problem that is now fairly widely recognized; when we were chatting offline before we went on camera, I mentioned to you that I attended this conference two years ago and I didn’t see very much discussion about that particular phenomenon; now in several of the sessions that I’ve been attending, it has come up, although it comes up in a technical audience where the participants are all project leaders and techies; so it’s very nice for the speaker to say yes, we need to get the CEO and the CIO involved, but they’re not here; so I’m not quite sure how it’s going to come about, although obviously we are beginning to see some companies especially the high tech companies being run by recent or former or even current techies.
So I should have mentioned by the way that when I asked all the CIOs about Agile, the CIO of Google really understands what it means: he’s 28, and Google as a culture I think is very Agile not just the day-to-day operational activities of the project teams but all the way up to the top. So I think you are going to see it gradually happening, but it may take another generation particularly for the companies that have been around for a hundred years that have such a deeply ingrained kind of command and control mindset; it’s encouraging to see that it’s even recognized as the next major battle that needs to be fought but I think it’s going to take a while.
7. So just because we’ve had ten years of Agile doesn’t mean we’re there yet?
It might mean that we’re there from a tactical point of view, although even that is still debatable; there was a session the night before last I think with a bunch of the industry analysts, people from Gartner and Forrester and so on and one of the questions to the Gartner fellow was “two years ago in 2010 you predicted that by 2012, i.e. today, 80% of companies would be using Agile, has that happened?”; and the Gartner guy said “well I think 80% of companies are doing some Agile somewhere sometime but probably only for 40% of their projects”. So I think that even at the tactical level while Agile clearly is now mainstream, it’s not fully pervasive, it’s not fully deployed and we, I don’t know do we have another 2 or 3 or 5 or 10 years before you can say that that’s the case.
Someone remarked to me out on the conference area that he was discouraged by the number of projects that are starting up even today that are still waterfall and I said yes, “you’d also be kind of discouraged to see how many of them are still in COBOL”. Just because new ideas and new technologies and so on have finally caught on and are being used in a reasonable number of cases or companies doesn’t yet mean that the old stuff has died away; we have a lot of legacy and a lot of inertia and a lot of that is people and corporate cultures; until the current generation of project leaders and next level executives dies away, I think it may still be a battle for some of the Agile folks.
Shane : Culture shift is not easy.
Or quick, it’s not easy or quick.
I think there are a lot of things in common, although I think if I had to articulate one fundamental difference, 30 years ago we still had the feeling that if we just had better ways to do things, we could do it right the first time; so we can have perfect requirements and a perfect design, write a perfect code which will work the first time and live happily ever after; and I think the Agile movement has said “give up, don’t even try to do that because we are imperfect human beings living in a turbulent world blah, blah”; but it is interesting that 30 years ago we tried, my colleagues and I, tried to explain to people that just because we had these ideas about modeling requirements did not mean you really had to do it to the last excruciating level of detail before you actually performed something useful.
So a lot of the top down incremental implementation concepts that are very much at the heart of Agile today are things that we tried to communicate, but we didn’t really put it into our books very well; I mean I actually did put it into part of one or two books but not at the forefront and people certainly didn’t hear it; but even 30 years ago, we realized that things were going to change; that you could start with a set of requirements and somebody might change what they thought they wanted and so forth, it didn’t happen as fast and as abruptly as it happens today but it was still part of the equation.
Well certainly they can build on that experience; it’s very common for new ideas to be introduced, I mean this has happened several dozen times in various parts of the IT industry during my career and there’s this well-known technology adoption curve where the initial period is being popularized by the lunatic fringe who tends to promote their ideas based on essentially religious faith; “clearly all these other ideas are bad, anything we do will be better and these ideas I’m proposing make sense”, but you still had to follow that based on faith.
Now we’ve got ten years of experience with Agile where the people speaking today and it’s a little discouraging for me to see that the people I regarded as the young turks are now old themselves, but anyway they can say “we think these are the ways to do things and we‘ve got ten years of case studies and results and data; we’ve got real data from real projects that we can use to validate these”; that turns out be absolutely crucial. If you remember some of the technology adoption stuff, to get past that very critical period of going from the innovators to the mainstream; because you’ve got a bunch of projects and people and organizations that are frustrated and discouraged by what they’re doing now but before they take the leap to something new whatever it is, they want to see some evidence which inevitably means they’ve got to see results from somebody else who is brave enough to do it on faith and that at least we’ve got.
So I think where we are right now in terms of Agile is a situation where we’ve got a pretty good idea of what needs to be done technically or operationally to get project teams working with customers to develop good systems; we’ve got the vocabulary, we’ve got the checklists; we’ve got the documented procedures, we got a whole bunch of tools that help support it - and that’s all great, but there’s still the human side, the social dynamic of getting a bunch of people in the room who actually can collaborate and cooperate and join forces to build something; you don’t change human nature overnight just like you don’t change corporate cultures overnight. So I think there’s also a shift towards focusing on that; and the people who were brilliant at articulating and popularizing some of the technical issues are not necessarily the people who have the sociological or psychological or anthropological background that can really address some of the human issues.
10. So if we think back to some contemporaries of yours, who is the modern Gerry Weinberg?
I think that’s a good question; now there may be some people here at this conference who would say “I am, I am”; in fact the one person who comes to mind that I listened to a day or two ago is Linda Rising; she is very proud of telling us that she’s 70 years old and that she’s gone through several careers and has accumulated a bunch of degrees and knows and awful lot about culture; and her session was kind of a Q&A thing and it was amazing how many people got up on the sofa to talk to her and ask about how to deal with their teenage children; and maybe the parallel was that a lot of the people in your project team behave like surly teenagers sometimes, which I certainly have experienced. So there probably are some and I’m probably remiss in not giving a list of two or three but I don’t think of any of the giants like Gerry Weinberg coming to mind right away.
Some of his cohorts by the way are here in this conference I know that but I think he himself is not here.
11. No, he’s not; so where do we go from here; where is the information technology industry moving?
Where we go from here certainly in terms of Agile I think is in two areas that we’ve already discussed; one is dealing with the human side of the equation so that we can get teams to collaborate and so forth; and even from an operational point of view, there is still the deployment, getting it throughout the organization which some of these speakers at this conference that refer to as “scaling” and I think they’ve got a good point that you get so called pockets of excellence - little individual project teams that are doing fantastic Agile stuff and now either they or maybe some higher level manager says, hey we really ought to spread this throughout our entire universe and there are good ways and bad ways of doing that and I think a lot of the people probably will fall flat on their face in their attempt and it’ll take another five or ten years probably and that is what will lead the Gartner person to say “my god, it is being used in 80% of the projects in 80% of the companies, it really is pretty well universally deployed at this point”; so we still got that issue to deal with.
And part of that as we’ve also discussed this morning is going to involve propagating the mindset and culture up the chain of command and I think ultimately that’s going to be a combination of some bottom up proselytizing and rabble-rousing and so on, combined with some top down stuff. We’re going to have to have enough gifted, articulate, respected top level people; it could well have been the Steve Jobs if he were still alive or the Bill Gates if he were still operationally involved (although I’m not sure he believes in Agile), but you’ve got to have some people with those kinds of reputations who have seen the light and who are going to come preach to their colleagues about the importance of operating in that way; that will keep us busy for another ten years I think.
That’s an interesting question because I think we’re going to see more and more of that as time goes on; I think to some extent this conference is still in kind of the minority situation in terms of most people talking about how to do Agile inside a company with internal logic teams and so on but the larger industry trend I think is more and more towards outsourcing; companies don’t want to have huge development teams, they really want to have however, IBM or Accenture or Tata or somebody else, Infosys, do the work.
Most of the consulting vendors that I’ve seen at this conference were relatively small organizations, who I assume are tackling relatively smallish to medium sized projects where they can probably find the cooperative client and say “we’re going to send all five of our people or all ten of our people to live in your world to do this Agile project”; I’m not sure how it’s going to take place on a larger scale and I think it’s going to require some significant changes in the whole contracting kind of mechanism that operates today. Because in my experience especially dealing with the lawsuits that eventually come out of these things, those are still very much waterfall oriented in terms of formal checkpoints and sign-offs and progress payments and things of that sort and the immediate tendency to blame the other side if something changes in the middle after “hey you agreed, you promised, you swore that this is what you wanted”, so I don’t know how that’s going to change.
It may change by virtue of some of these smaller vendors that we see in the convention today becoming so successful that as they grow, they can continue to insist on that kind of operational mode; “we’re not going to take on this project of yours unless you agree to do the following ten things that will enable us and you to proceed in an Agile fashion”.
13. And what about the big vendors, the Accentures?
I don’t know about the big vendors; I mean they’ve obviously all heard about Agile and as I’ve said earlier they all tend to put Agile buzzwords into their proposals; and certainly they will carry out the Agile part of the narrow implementation activities if they believe that the requirements are stable, it makes a lot of sense for them to do continuous testing and automated testing and continuous integration and blah, blah, blah, blah, blah; but on a larger scale, I don’t know what will force them to change; it may be a severe competitive threat or maybe they won’t change; we’ve certainly have seen situations before where a large vendor just sort of fade away because they’re too tied to obsolete ways of doing things; it remains to be seen.
Shane : It does indeed; Ed, thank you so much for taking the time to talk to InfoQ today; we really appreciate it.
My pleasure.
Shane : Enjoy the rest of the conference.
Thanks very much.