I am searching for some references that show that Agile projects have a higher chance of success than other approaches.The references should be quotable in a government document ie. come from a trustworthy source.
Studies are a powerful way to convince skeptics of the value in Agile and Lean methods. Conversely, the lack of strong, documented evidence of success can drastically weaken a case for adoption of Agile and Lean methods within an organization. Luckily, there are some studies out there, summarized in this article.
An extraordinary list of references to Agile studies, XP studies, and related material can be found on the reference list of the book Succeeding With Agile by Mike Cohn.
A highlight of the Succeeding with Agile reference page is a superb and concise set of results at Agile Adoption Rate Survey Results: February 2008. This survey results page is on Scott W. Ambler's Agile surveys site, which contains a wealth of information relevant to evaluating Agile versus Waterfall and other methodologies, and evaluating specific Agile techniques against each other.
Many studies of Agile effectiveness can be found on a page called, unsurprisingly enough, "Studies of Agile Effectiveness." This page contains links to case studies that illustrate both the positive and negative aspects of Agile software development.
The Studies of Agile Effectiveness page is hosted by George Dinwiddie on his Agile Bibliography wiki. The wiki is modifiable by the public, and Dinwiddie has requested that anyone who has links to new studies to add them to the page.
An article by Michael Mah and Mike Lunt entitled, "How Agile Projects Measure Up, and What This Means to You" directly addresses the measurement of Agile effectiveness. From the abstract:
In this Executive Report, Michael Mah and Mike Lunt share observations about real-world agile projects and how they measure up against waterfall and other projects in terms of productivity, schedule, and quality. This was made possible by juxtaposing them against trends from a contemporary worldwide database of more than 7,500 completed projects. Specifically, we look at more than 20 agile releases from five companies. Two of the companies are examined in specific detail, each having achieved best-in-class levels of performance by implementing agile in very different ways.
For information about the effectiveness of Lean, it's hard to beat the books that popularized Lean in the English-speaking world: The Machine That Changed the World, and Lean Thinking. These books have numerous case studies and lots of data about the success of Lean techniques within the manufacturing world.
Lean software development is in many ways different from Lean manufacturing, however. Case studies that are about Lean software development rather than about Lean manufacturing can be found in the paper "Lean Software Development: Two Case Studies" by Peter Middleton.
One disadvantage of many of these studies is that they lack peer review, leaving them open to the criticism that they lack the intellectual rigor necessary to prove the effectiveness of Agile techniques over Waterfall or other methods. Mark Woyna makes the point, however:
Does anyone have a link to peer reviewed studies that prove that Waterfall projects are more successful than process X, or that RUP projects are more successful than Waterfall projects?Anyone needing that level of "proof" is simply looking for an excuse not to try agile.