Six Agile Method Take Aways from the Reifer 2014 Quantitative Analysis of Agile Methods Study

Six Agile Method “Take Aways”

Reifer Consultants LLC recently published a benchmarking report that compared the productivity, cost and quality performance achieved by software development projects that use agile methods against similar ones that employ traditional, plan-driven approaches. The results of the analysis were based on 1,500 projects, 500 of which employed a variety of agile methods, over a ten year period using data supplied by 100 organizations. This condensed white paper summarizes seven ‘trends and take-aways’ taken from our report entitled “Quantitative Analysis of Agile Methods¹.” A more comprehensive version of this ‘take-aways” white paper is available here for a fee.

Take Away - Agile Adoption Increasing

Agile adoption continues daily with approximately sixty-five percent of the software development work currently in progress in the 100 organizations polled being agile. This includes agile at scale projects which represent about 125 of our 500 project agile population. These agile at scale project involve coordinating deliveries made by teams of more than 75 professionals that are developing software products across geographically-dispersed locations. Unfortunately, only about 10 percent of the systems and digital engineering work done in those firms building products that include hardware as well as software is being performed using agile inspired processes. This lack of a single agile engineering process limits the effectivity of the approach and, of course, its benefits.

Take Away - Agile Productivity

Agile productivity, as measured in terms of outputs/unit of input, continues to be about 10 to 25 percent higher than the norms experienced on plan-driven or traditional projects for products delivered during 2014. The average gain in productivity experienced for those using agile methods continues to be at about 13 percent per year. Gains in productivity for those using agile methods have ranged from 10 to 25 percent each year during the past ten years. As in the past, these gains vary by application domain and those factors that influence the software productivity ((product complexity, processes utilized, size of project, degree of automation, etc.).

Root Causes: improved teamwork, use of lightweight processes, focus on generation of working code, and possibly the Hawthorne effect⁴.

Issues: the top five current concerns relative to productivity include:

• Process mismatches abound between agile and project management infrastructure practices.
• Emphasis on dogma instead of practicum when trying to put agile into practice.
• Contracting concerns over agile emphasis on time and materials contracts.
• Product versus project management confusion.
• Measures of value that drive agile developments are vague and non-quantitative.

Exploitation Measures: some suggestions for exploiting agile productivity gains include:

• Understand the variables that impact productivity (architecture stability, product complexity, staff skills and experience, etc.) and address them as you move to the use of agile.
• Place attention on automation and regression testing where tools are needed to revalidate releases once changes have been made to them.
• Gather productivity measures and metrics and use them to help make needed adjustments.
• Reduce resistance to change when moving to agile by showing critics that agile represents a better way of doing business using the data collected on your projects.
• Once the transition is made, maintain operations tempo and momentum by expanding the use of agile methods in aligned areas like systems engineering.
• Focus agile on developing those applications where they can realize their value proposition

Take Away - Agile Cost 

Agile costs, as measured in terms of $/unit of output, continue to be about nine percent less per year than the traditional norms with an average cost of $3/eSLOC less than that being observed for plan-driven projects. The reductions in cost experienced by users of agile methods range from 10 to 20 percent per year during the past ten years. As with productivity, agile expenditures vary by application domain and those factors that influence project costs (teamwork, workforce composition, overhead rates, human resources policies, etc.).

Root Causes: highly motivated teams, reduced management overhead and reporting, lower labor rates, simplified communications patterns, and a product rather than process focus.

Issues: the top five current concerns relative to cost include:

• Roles and responsibilities of key players (product owner, team members, project manager, etc.).
• Investments in infrastructure, facilities, equipment and tools for agile.
• Mismatches with supporting processes (requirements engineering, quality assurance, project management, configuration management, human resources, etc.).
• Degree of subcontracting and out-sourcing along with related management practices.
• Time required to transition to agile practices.

Exploitation Measures: some suggestions for exploiting agile cost advantages include:

• Understand the variables that impact cost (they differ from productivity) and address them as you move to harvest cost reductions.
• Time-box the project and then develop overall estimate for the project before allocating time and labor to individual sprints (see our Estimating Guide⁵).
• Buy the right tools (agile metrics toolset, regression test suite, etc.) and setup the proper facilities (war room, task-board, etc.) prior to starting.
• Monitor and control costs on the project and its position relative to the budget.
• Do things as cheaply and simply as possible (both process and product).

Take Away - Agile Time-to-Market

Agile time-to-market, as measured in terms of calendar time from start to software delivery, averages between 10 to 50 percent less than the norms being experienced on plan-driven or traditional projects, which is at best a reduction of between 20 to 40 percent in a single year based on size of the project when delivered. However, not all features that were promised were always delivered. The philosophy observed is to deliver a working software product with those features that the customer/user considers important. Other non-essential features are backlogged and handled with later deliveries. In contrast, traditional developments adhere to an “all or nothing” philosophy where delays rather than timely deliveries are the norm.

Root Causes: focus on incremental delivery of working software, fast-paced process, continuous development in sprints, and delivery ending with demo versions before the final hits the market.

Issues: the top five current concerns relative to cost include:

• Whether the original schedule and budget were reasonable.
• Whether the schedule could be time-boxed to ensure that the important features (i.e., those that contribute directly to the value proposition) were delivered first.
• The percentage of features that are considered acceptable at delivery.
• The number and priority of features in the backlog that are considered acceptable at delivery.
• The number and type of defects in the backlog that are considered acceptable at delivery.

Exploitation Measures: some suggestions for exploiting agile time-to-market advantages include:

• Understand the variables that impact time-to-market and address them as you move to agile.
• Prepare duration estimates for the project based on content and adjust schedules to meet them.
• Monitor and control milestone achievements as you develop the product.
• Let the team control timelines for sprints.
• Set expectations for increment timelines and content.
• Hold demonstrations at the end of each increment to show-off a working product and engage customer/users, marketing and management.

Take Away - Agile Quality

Agile quality, as measured in terms of defect densities post-release, averages about 6 percent better than that being experienced on plan-driven projects. It needs to be noted that quality at delivery when first starting to use agile methods is lower by as much as 20 to 30 percent as compared to traditional methods. However, quality quickly catches up after three years and then gets as much as 5 to 10 percent better once the transition to agile methods has been fully made⁶. Again, quality varies greatly by application domain and those factors including those factors that influence quality including quality control and assurance practices.

Root Causes: overcoming the fear of the process police, setting quality goals for product, placing early emphasis on engineering quality into the product, and establishing a quality culture to guide decision-making throughout the life cycle.

Issues: the top five current concerns relative to software product quality include:

• Quality assurance equated to testing in many agile shops.
• Rush to create working software often done with lack of early emphasis on quality.
• Quality metrics neither gathered nor used to make necessary improvements during development.
• Lack of use of quality control and assurance practices.
• No independent look at the product to ensure quality expectations are realized before release.

It should be noted that confusion over how to apply agile methods occurs in a regulated environment over who should verify software satisfies safety, security and other such governance requirements.

Exploitation Measures: some suggestions for maintaining high software quality standards follow:

• Transform the culture to one that rewards quality rather than timely delivery.
• Embrace quality and make it part of all of your work processes. Hold peer reviews, discuss quality at your daily standups and make defect information on a daily basis on your task-boards.
• Task everyone on the team to check the quality of each other’s products. Make it fun by holding contests, giving prizes and providing recognition for a job well done.
• Capture quality metrics and measures⁷ and use them to quantify quality throughout the life cycle.
• Never release a product that is defect-prone. Walk the talk and take the heat when conflicts between quality and delivery occur.
• Employ lean and Kanban⁸ principles which focus on engineering quality into the product as it is being developed. Find and fix defects during sprints. Else, put them in the backlog.

Take Away - Agile Value

Agile methods place emphasis on providing its customers through early and continuous delivery of valuable software. Value is measured in terms of the financial benefit that organizations receive for their expenditures. However, little guidance is typically provided by agile advocates on how to quantify the value derived via their efforts. To address this issue and demonstrate the business case for agile methods, we recommend using classical measures such as cost/benefits, payback analysis, break-even analysis, or return-on-investment to quantify the value of the agile alternative because these measures are well understood and used for decision-making by upper management⁹.

Root Causes: talk about value without quantifying it in financial terms and ability to determine the true costs and benefits of using agile methods.

Issues: the top five issues relative to portraying the value of using agile methods include:

• Value propositions tied to current customer business goals
• Value propositions formulated in technical rather than economic terms.
• Not taking the customer’s current financial goals and infrastructure into account when developing the numbers.
• Not expressing the numbers in terms that the customers understands and needs to make the case for agile methods.
• Value propositions not able to hold up under scrutiny when inspected by outsider examiners.

Exploitation Measures: some suggestions for quantifying value follow:

• Whenever possible, let your numbers do the talking.
• If possible, use money as the common denominator when develop value propositions.
• Make sure that you quantify both the tangible and intangible benefits.
• Ensure that you consider both recurring and one-time costs.
• Consider sunk costs irrelevant because they have already been incurred
• Be sure to take into account the cost of money when dealing with financial people.
• Take existing financial folklore into account when expressing your numbers as these figures will be the ones that management will use to determine the reasonableness of your results.

Major Findings

Because agile methods represents a new way of business for many of the organizations surveyed, the “take aways” discussed in the previous paragraph should be expected. Many of the issues highlighted like agile scaling, maintenance and contracting continue to plague organizations that have made the transition. Once transition has been accomplished, firms seem to have difficulty building an operational tempo that allows them to continue to tap the many benefits of agile methods. The goal should be to maintain momentum as they transition the methodology into widespread use. Techniques exist to achieve this that revolve around developing and putting best practices to work within a collaborative team environment that embraces interdisciplinary approaches and a product management focus and discipline.

Based on everything we learned based on analysis of hundreds of agile projects during the past decade, the following six findings continue to be pertinent to those trying to harvest the benefits of agile methods:

1. While there is a definite business case that can be made for the use of agile methods, there are serious barriers to adoption that need to be overcome by newcomers to exploit use of the methodology.
2. Change management principles¹⁰ dominate as you transition to using agile methods. When making the move, strategies need to developed, plans put into place, infrastructure revamped, staff trained, management educated and pilots run to figure out the best way to run agile projects within the context of your organization, its business goals, its processes and its past history. In addition, measure your progress and use metrics to help guide you on your path to success. One good way to avoid the traps that have snared others in the past is to bring in an agile coach and mentor as part of your team.
3. You should recognize that some agile methods work and others may not for different types of software development projects. For example, Scrum works well for small to medium sized commercial projects and scaled agile methods like SAFe¹¹ seem to work well for larger ones. Other lightweight methods like the Rational Unified Process (RUP)¹² and the Team Software Process (TSP)¹³ seem to be better suited for projects that are safety critical (air traffic control, nuclear power plant operations, etc.) and others where a high degree of governance (financial systems, etc.) is required. As an additional consideration, firms need to attempt to integrate agile approaches with those used as the management and support mainstays of the firm.
4. Once widespread transition has been made throughout the software organization, firms try to maintain the momentum by building an agile operational tempo. As they switch gears, they try to spread the methodology into widespread use throughout the organization and into aligned engineering and support groups. While this may seem simple, it is not. Groups like systems engineering and test tend to resist the change because they view agile approaches as undisciplined. As expected, considerable time and effort and some executive encouragement is needed to nudge agile adoption ahead in all of these areas. The major two lessons learned are that these groups should be brought on-board as soon as possible and that a single engineering methodology should be embraced right from the start.
5. Scaling of agile projects continues to be a problem because of organizational and process mismatches and the confusion over whether the emphasis should be either project or product management. When conflicts between groups exist, little gets done. To avoid these issues, we recommend moving to a single engineering discipline that embraces affected groups, exploits true interdisciplinary team concepts, revamps processes with agile concepts in mind and takes full advantage of project management techniques like those employed by the Project Management Institute in their Body of Knowledge (PMBOK)¹⁴. In this case, a good management expert who possesses agile, process improvement and project management skills, knowledge and abilities can be of assistance especially if he/she has experience within your industry and applications domain in which you operate.
6. From the competitive landscape, the move to agile methods often makes a lot of sense for three reasons. First, you can use the methodology to catch up with your competitors who have most likely already made the commitment and made the move to use agile methods. Second, you can use agile methods as a means to attract talent. The reason for this is that high performers want to work in an environment that uses the latest and greatest techniques. Third and finally, you can use agile to show your customers that you are adopting techniques that provides them with value; i.e., increase productivity, cut cost, reduce time-to-market, improve quality and provide a competitive advantage.

For those of you interested in learning more about agile and its benefits and weaknesses, we refer you to our web site to view our blog, news flashes and current reports which are summarized for your convenience in Table 1 of this document.

References

¹ Reifer Consultants LLC, Quantitative Analysis of Agile Methods, No. 8, v1, 1 July 2014.

² Kenneth Rubin, Essential Scrum: A Practical Guide to the Most Popular Agile Process, Addison-Wesley, 2012.

³ Sondra Ashmore and Kristine Runyan, Introduction to Agile Methods, Addison-Wesley, 2014.

⁴ Elton Mayo, Hawthorne and the Western Electric Company, The Social Problems of an Industrial Civilization, Routledge, 1949.

⁵ Reifer Consultants LLC, Agile Estimating: Straightforward and Simple, v1.4, June 2014.

⁶ See our web site.

⁷ Reifer Consultants LLC, Agile Software Quality - A Quantitative Analysis, October 2013.

⁸ Henrik Kniberg, Lean from the Trenches: Managing Large-Scale Projects with Kanban, Pragmatic Bookshelf, 2011.

⁹ Donald J. Reifer, Making the Software Business Case: Improvement by the Numbers, Addison-Wesley, 2001.

¹⁰ Donald J. Reifer, Software Change Management: Case Studies and Practical Advice, Microsoft Press, 2011.

¹¹Dean Leffingwell, Scaled Software Agility: Best Practices for Large Enterprises, Addison-Wesley, 2007.

¹²Ahmad K. Shuja and Jochen Krebs, IBM Rational Unified Process Reference and Certification Guide: Solution Designer (RUP), IBM Press, 2008.

¹³ Watts S. Humphrey, TSP: Leading a Development Team, Addison-Wesley, 2005.

¹⁴ See this.

About the Author

Donald J. Reifer is recognized as one of the leading figures in the fields of software engineering and management with more than forty years of progressive management experience in both industry and government. In the 1980’s, he founded two software firms which were market leaders at the time. Since then, he served a program manager and executive in industry, an executive director in government, and a management consultant. His agile credentials include successful managing several large agile transformation projects and over ten years as an agile executive coach and metrics guru.

Appendix A - Available Reports - Reifer Consultants LLC

+ For current availability, pricing and more information, please see this.