The CHAOS statistics on project failure are frequently cited by those implementing Agile or other forms of process change, so their validity is of importance to the many people who continue to use them. In August, triggered by Robert Glass's questions in August's Communications of the ACM , InfoQ interviewed Jim Johnson, creator of the CHAOS Chronicles on project failure, to hear how their data is collected.
In the discussion thread that followed the interview, one question surfaced as still unresolved: how does the Standish Group explain the amazing improvement in cost overrun from 189% in 1994 to 69% in 1998? The implication of this question, also posed in several other places on the web, is that Standish may have adjusted their research methods without revealing the change in their reports. If true, comparisons of some CHAOS data from 1994 and 1998 could be meaningless.
Standish founder Jim Johnson was also concerned by the results he saw between 1994 and 1996, so concerned in fact that the 1996 survey was never published. In this article he shares what their research revealed about major changes in the world of software development in the mid-nineties, which significantly changed the complexion of project planning and execution. InfoQ brings you an excerpt from this month's CHAOS University newsletter.
Cosmic Intergalactic Change
Excerpt from the November 2006 CHAOS University Newsletterby Jim Johnson, Founder and Chairman of The Standish Group
Lately a few people have asked us to explain the amazing improvement in project cost overruns, from 189% in 1994 to 69% in 1998. We have done these explanations on many occasions and in many venues. So, what I am going to say is not new. It is included in both of our recent writings, the CHAOS Chronicles 3.0 and our new book, My Life is Failure, but a direct explanation may not jump off the pages and hit you in the face. So here goes.
It would be both great and self-serving to say that the improvements were the result of our research, but we cannot take credit for such improvement. On the other hand, we did see in many of our focus groups and workshops after our initial published report that user involvement was up, executive support was greater, and people were clearer on their business objectives. Also, many organizations are quickly moving to have their project managers PMI certified. All this adds to improvement, but we doubt it would move the results but a few points. This might have resulted in the improvement from 1994’s 189% to 1996’s 142%.
In this month's DARTS survey we asked SURF participants to rate their skill level on estimating project time and cost. A little less than ten percent thought they were very skilled, while almost two-thirds thought they were moderately to poorly skilled.
How skilled you are at estimating time and cost has a direct impact on time and cost overruns. During our spring focus groups of 1998, we asked our IT participants if they had changed the way they estimated projects. The consensus was, they used to estimate the project and then add a safety amount for contingencies and other items for a more conservative approach. In 1998, they had changed their process so that they were then taking their best estimate, and then doubling it and adding half again. Such a change in estimating procedures would directly result in the improved cost overruns, as we saw from 1994 to 1998. However, it is our opinion that this is not what caused the overruns to dive from 189% to 69%. So, before you read the next paragraph, I want you to think back on what was going on between 1994 and 1998. Lean back, close your eyes, and think what cosmic intergalactic change happened to the IT industry and computer applications that were developed at that time.
Eyes open now? How about your mind? First a clue! In 1996 we showed a mammoth forty 40 percent of projects were canceled. In fact during that year we were so devastated by the results you will not find a formal 1996 CHAOS report. Nonetheless, if you look at any CHAOS table that contains all the years and you will see these results.
What happened in 1996 to cause so many failures? The answer is the Internet. We went from a client/server development and implementation style to an Internet style. Client/server applications are a magnitude more complex to develop and implement than Internet applications. They would always get screwed up, and you never knew what was in the user’s PC that would cause conflicts. It was a mess, and organizations dumped their client/server projects as fast as they could. Now with Internet-style development, projects are much smaller, simpler, faster to implement, and easier to manage. This push to browser-based clientless access has been the catalyst to this phenomenon. In addition, now we see that IT has the liberty to use their new way of estimating. Bang, bang – improvement! While we have seen steady improvement since 1998, we think it will take another cosmic intergalactic event like the Internet to see these types of improvements again. However, the lessons learned from this single event are immense and drives much of our thinking on the proper way to develop software.
