Author and consultant Scott W. Ambler has conduct an annual survey examining IT project outcomes since 2007. He makes the survey responses and all his evaluation freely available so others can conduct their own analysis of the content.
He summarised the 2011 survey results in a Dr Dobbs article titled "How Successful are IT Projects, Really?".
He explored the impact of "development paradigm" on the outcome of projects, with a view to identifying which approaches work best. He describes the five approaches he examined as:
On an ad hoc software development project, the team does not follow a defined process.
On an iterative software development project, the team follows a process that is organized into periods often referred to as iterations or time boxes. On any given day of the project, team members may be gathering requirements, doing design, writing code, testing, and so on. An example of an iterative process is RUP.
On an agile software development project, the team follows an iterative process that is lightweight, highly collaborative, self organizing, and quality focused. Example of an agile process is OpenUP, Scrum, or XP.
On a traditional software development project, the team follows a staged process where the requirements are first identified, then the architecture/design is defined, then the coding occurs, then testing, then deployment. Traditional processes are often referred to as "waterfall" or simply "serial" processes.
Lean is a label applied to a customer value-focused mindset/philosophy. A lean process continuously strives to optimize value to the end customer, while minimizing waste that may be measured in terms of time, quality, and cost. Ultimately, the lean journey is the development of a learning organization. Examples of lean methods/processes include Kanban and Scrumban.
When analysing the results he used the following definitions for successful, challenged and failed projects:
In the case of this survey, a project was considered successful if a solution was delivered and it met its success criteria within a range acceptable to the organization. A project was considered challenged if a solution was delivered, but the team did not fully meet all the project's success criteria within acceptable ranges (e.g., the quality was fine, the project was pretty much on time, but ROI was too low). A project was considered a failure if the project team did not deliver a solution at all. In future surveys, I think I will rework the failed definition to exclude projects that were purposefully cancelled with the full agreement of the stakeholders.
Using these definitions he found the following results:
- Iterative and agile approaches had statistically the same success rates, with 69% of iterative projects being successful and 25% challenged; whereas 67% of agile projects were considered successful and 27% challenged.
- Traditional and ad hoc teams also statistically had the same levels of success, which in turn, were statistically lower than those enjoyed by agile or iterative teams. In this case, traditional teams were 50% successful and 36% challenged, and ad hoc teams were 49% successful and 38% challenged.
- 62% of lean projects were considered successful and 30% challenged, putting it dead center between the agile/iterative and traditional/ad hoc pairings.
The article goes on to examine the relationship between various factors that contribute to project success and the approach used. He looks at success as a multi-dimensional construct:
- Time/schedule: 20% prefer to deliver on time according to the schedule, 26% prefer to deliver when the system is ready to be shipped, and 51% say both are equally important.
- Return on investment (ROI): 15% prefer to deliver within budget, 60% prefer to provide good return on investment (ROI), and 25% say both are equally important.
- Stakeholder value: 4% prefer to build the system to specification, 80% prefer to meet the actual needs of stakeholders, and 16% say both are equally important.
- Quality: 4% prefer to deliver on time and on budget, 57% prefer to deliver high-quality systems that are easy to maintain, and 40% say both are equally important.
He compares the outcome of each dimension to the development approach used, and finally concludes:
Once again, this survey has shown that agile and iterative software development methods appear to be more successful than either traditional or ad hoc strategies. This is consistent with previous surveys and is likely part of the explanation for the general trend in the IT community toward adopting agile strategies
Ambler is definitely not the only one conducting project success surveys. The most commonly cited is the Standish Group Chaos Survey, which uses a much more restrictive definition of success as on-time, on-budget and on-scope. The latest Standish Group Chaos Report was released in March 2011. Standish chairman Jim Johnson stated
"This year's results represent the highest success rate in the history of the CHAOS Research, We clearly are entering a new understanding of why projects succeed or fail."
Is Ambler's definition of success valid, and do his results reflect the reality in your organisation?
Participation in this survey is self selected - does that influence the results?