Ten ‘Take Aways’ from the Reifer “Quantitative Analysis of Agile Methods” Study

I. Ten Agile Method “Take Aways”

Reifer Consultants LLC recently published a benchmarking report entitled “Quantitative Analysis of Agile Methods¹” that compares agile method software productivity, cost, duration and quality to that of traditional, plan-driven projects. The results of the analysis were based on 800 projects, 250 of which are agile, over a ten year period using data supplied by 60 organizations on their completed efforts.

The following ten “take aways” can be gleaned from our agile study whose results are documented in the referenced benchmarking report. The report is currently being sold for $795 and can be ordered from here and the International Software Standards Benchmarking Group here.

1. Agile productivity - Agile productivity, as measured in terms of outputs/unit of input, is higher than the norms experienced on plan-driven projects for delivered products. The gains experienced range from near nominal to 50 percent over the course of ten years, averaging at best between a 10 to 20 percent improvement in a single year after adoption. However, these gains vary greatly by application domain and are a function of many factors (workforce composition, product complexity, size of project, etc.). For example, agile methods seem not to make sense for projects that need some form of certification (i.e., for flight safety and security certifications). Capers Jones’s data² confirms ours by suggesting that such projects go through certification more quickly when using more structured process like the Rational Unified Process (RUP)³ or the Team Software Process (TSP)⁴.

Root Causes: improved teamwork, use of lightweight processes, focus on product and working code and possible Hawthorne effect.

Issues: development or maintenance, organizational issues, process rigidity and dogma, overly strict degrees of governance, contracting problems and lack of product management focus.

Exploitation Measures: some suggestions for exploiting agile productivity gains follow:

• Understand the variables that impact productivity (architecture stability, product complexity, staff skills and experience, etc.) and address them as you move to harness agile.
• Gather productivity measures and metrics and use them to help make needed adjustments.
• Beat down resistance to change by showing that agile represents a better way of doing business using the metrics and measures collected on a pilot project.
• Adapt your processes and agile methodology in lock step using the pilot project to find the right path to achieve your business goals.
• Avoid governance and contracting issues on first projects by doing the project in-house, whenever possible.
• Focus agile on developing those applications where it has and can make a difference.
• Be successful with agile as success breeds more success.

2. Agile cost - Agile cost, as measured in terms of $/unit of ouput, is less than the norms being experienced on plan-driven projects. The gains experienced range from near nominal to 100 percent over the course of ten years, averaging at best a cost avoidance of between 20 to 40 percent in a single year after adoption. Again, these costs vary greatly by application domain and are a function of many factors including those revolving around workforce composition and labor rates.

Root Causes: lower labor rates, less management overhead, simplified communications patterns and a product rather than process focus.

Issues: development rather than maintenance cost avoidances, organizational realignments and role clarifications, process rigidity, overly strict degree of governance, contracting problems and a lack of product management focus.

Exploitation Measures: some suggestions for exploiting agile productivity gains follow:

• Understand the variables that impact cost (they differ from productivity) and address them as you move to harness agile.
• Be realistic in your estimates for the project (the sum of the sprints often is an underestimate of what it will take to deliver the end-product).
• Monitor and control costs on your project and keep track of where you are relative to budgets.
• Focus early spending on education and training for agile.
• Buy the right tools and setup the proper facilities (war room, task-board, etc.) prior to starting.
• Do things as cheaply and simply as possible as you move to the use of agile.

3. Agile time-to-market - Agile time-to-market, as measured in terms of calendar time from start to software delivery, is between 10 to 60 percent less than the norms being experienced on plan-driven projects, averaging at best a reduction of between 20 to 50 percent in a single year based on size of the project after adoption. However, functionality and features delivered do not always match either customer or the user’s requirements and expectations as they are often omitted to accelerate delivery time. This leads to disappointments.

Root Causes: focus on developing working products, fast-paced development in sprints and delivery of increments ending with demo versions before the final hits the market.

Issues: it is OK to end sprints with less functionality than promised, backlogs may not be depleted, customer/user may not always be a member of team and management may not always be engaged.

Exploitation Measures: some suggestions for exploiting agile productivity gains follow:

• Understand the variables that impact time-to-market and address them as you move to agile.
• Develop realistic duration estimates for the project as a whole based on content and adjust sprint and increment schedules to meet them.
• Monitor and control milestone achievements as you develop the product.
• Let the team control timelines for sprints.
• Set expectations for increment timelines and content.
• Hold demos at the end of each increment to show-off a working product and engage customer/users, marketing and management.

4. Agile quality - Agile quality, as measured in terms of defect densities post-release, is between 2 to 8 percent higher than that being experienced on traditional plan-driven projects, with quality at delivery off by as much as 20 to 30 percent. Again, quality varies greatly by application domain and is a function of many factors including those revolving around quality control and assurance practices.

Root Causes: fear of the process police, too much emphasis on test and not enough on quality throughout the project, few quality goals set for product, quality not engineered into the product and lack of quality culture to guide decision-making.

Issues: rush to code in agile creates lack of early emphasis on quality, quality assurance equated to testing, quality control not performed, quality metrics not gathered and lack of use of proven quality control and assurance practices.

Exploitation Measures: some suggestions for exploiting agile productivity gains follow:

• Transform the culture to one that rewards quality rather than timely delivery.
• Embrace quality and make it part of all of your work processes. Hold peer reviews, discuss quality at your daily standups and make defect information gathered from your version control repositories visible on a daily basis on your task-boards and version status reports.
• Task everyone on the team to check the quality of each other’s products. Make it fun by holding contests, giving prizes and providing recognition for a job well done.
• Capture quality metrics and measures and use them to quantify product quality during both software development and maintenance.
• Always perform “root cause” analysis for quality problems to avoid focusing on the symptoms and not the cause of the issue triggering the defect.
• Never release a product that is defect-prone. Walk the talk and take the heat when conflicts between quality and delivery occur.
• Employ lean and Kanban⁵ principles which focus on engineering quality into the product as it is being developed. Find and fix defects during sprints. Else, put them in the backlog.

5. Agile competitive advantage - Agile represents a competitive advantage to those organizations that can take advantage of its strengths and compensate for its weaknesses. Not only does it provide them productivity, cost and time-to-market advantages, it provides them with an ability to attract and retain talent which is the intellectual capital that can be the differentiator in competitive marketplaces. However, do not use agile methods in applications domains where they do not function well and do not make sense (applications requiring certification; large, distributed, complex developments, etc.).

Root Causes: agile has been over-hyped and over-sold, anything new is viewed with suspicion and many firms view software as not part of the competitive landscape (i.e., it is a support service rather than a revenue producing activity).

Issues: agile weaknesses need to be exposed as well as its strengths, organizational changes are needed as well as new roles and responsibilities and the move to agile is full of dangers as is anything else that is new within an organization.

Exploitation Measures: some suggestions for exploiting agile productivity gains follow:

• Make those structural changes to the organization and the processes required to move agile into the mainstream in the organization.
• Define agile job descriptions, roles and team concepts formally through position descriptions. Then staff positions, hire agile coaches and educate and train the workforce.
• Make the cultural shift from a top-down, command-driven style of leadership to self-organizing, participatory teams at least within your software organization as part of your move to agile.
• Make required governance changes. Instead of placing oversight from above or external sources to ensure compliance, leave agile teams alone and trust them to get the job done even when there are outside requirements for certifications. However, do monitor achievements and step in when metrics and other indicators imply things are going badly.
• Work towards achieving enterprise goals like reducing prioritized backlog instead of focusing on projects that deliver products that benefit one customer or user at the expense of another.
• Focus on keeping things simple. This philosophy extends from the product architecture to the software product itself. In addition, this focus on simplicity also extends to the structure of the organization that is charged with generating deliverables.
• Transform organizational focus from achieving individual goals (perform assignment on schedule, etc.) to providing value to the organization and its customers through teamwork and focus on generating working products.
• Create a culture where both successes and failures are celebrated and people are rewarded for making decisions, taking risks and trying to resolve problems.
• Direct efforts towards making decisions under uncertainty where outcomes can be predicted, but not with certainty, by the team.
• Focus on creating a better way of doing business and a better place to work for individuals, enterprises and their customers.

6. Agile at scale - Scaling of agile methods for large projects continues to be difficult. Based on our data set (800 projects across 60 organizations, many of which committed to the use of agile as long as 10 years ago), agile methods are not used heavily on large and complicated projects. While there have been some notable successes with agile on large projects, software engineering practitioners seem handicapped by the lack of structure and guidance when it comes to large projects. The data we have gathered shows that agile projects perform under par when it comes to geographically-dispersed and multi-national projects whose size exceeds 100 in total staff count. In addition, our data and that of Capers Jones suggests that agile breaks down when there are over 100 users and customers because of conflicts over the vision of what the systems should do. Perhaps, the root cause of this dilemma is that unlike agile, most large projects do not go straight to the user. Instead, they are delivered to some integration and test group where they are subjected to all sorts of system testing (performance, usability, etc.) before they are fielded and put into operations.

Root Causes: for the most part, agile practices are geared to small teams and projects. While some methods have published guidance on scaling, there is mistrust of them because of their lack of focus on and use of traditional project management techniques. In addition, as mentioned, agile products are delivered to the customer/user, while larger systems undergo system integration and test prior to being fielded and put into operations.

Issues: inability to synchronize, monitor, track and adjust simultaneous efforts done by different teams in different locations as part of development, inability to track product accomplishments and milestone completions, process mismatches and customer expectation disappointments.

Exploitation Measures: some suggestions for exploiting agile productivity gains follow:

• Breakdown organizational barriers to taking advantage of agile concepts by adopting self- organizing, participatory teams. For example, matrix concepts might have to be abandoned because they may create too much conflict with program management.
• Embrace traditional project management planning, scheduling and control concepts and put them to work on agile projects at scale to set goals and track achievements. Run the project using classic approaches, while running sprints using agile techniques.
• Employ agile organizational and team concepts as the norm rather than the exception for software development. But, augment them to address multiple customers and users and to address issues of scale that are a normal part of large projects (communications, etc.).
• Provide recommended structure and guidance for management of agile projects at scale for teams to follow.
• Try to mix agility and discipline in everything you do when developing software.
• Recognize that systems engineering and test personnel are customers for agile deliveries. But, do not forget to include the real users/customers/buyers of the system in your in-house deliberations because these are the folks who will determine your success or failure based on the features, functionality and performance that you deliver to them when the job is finally completed.
• Define and capture metrics and measures to determine how well you are doing relative to benchmarks developed at sprint, project and organizational levels.
• Address process mismatches and issues as part of the guidance provided. Pay special attention to those practices used with agile for configuration management, distribution management, quality control/assurance, and supplier management.

7. Agile systems engineering - Systems engineering practices need to be updated to be more agile. Else, there will be major conflicts in the organization due to process mismatches. For example, the development of system requirements needs to be synchronized as software development progresses because systems engineering staff develops and allocates them to both hardware and software level-by-level. The natural tendency of systems is to play customer. However, they typically do a poor job at this because many of them have not been exposed to operational roles. As another example, the systems architecture also needs to be stabilized early in the process because it represents the platform on which the software will operate. Systems engineering often fails to do this even when they take the software engineering staff’s suggestions for service-oriented features and plug-and-play functionality. As a final example, test engineering staff, which often is a part of systems engineering, needs to embrace continuous integration and testing to ensure that the system satisfies its requirements as it is incrementally developed. Unfortunately, systems engineering will not accept software deliveries with backlogs because it often believes that content cannot be postponed from one increment to another. Test engineering also needs to work with the software group to set in place the facilities and tools needed to integrate and test the software as part of the system. Sometimes actual users/operators of the system and real equipment are needed to validate the system as it goes through shakedown tests prior to acceptance for delivery to the field and operations.

Root Causes: system engineering arrogance, separate systems and software cultures and processes, early focus on system architecting at the expense of systems testing and lack of truly interdisciplinary and integrated team concepts.

Issues: requirements for software provided late, systems interfaces poorly defined, systems test focus at end instead of start of program, systems test and integration facilities late, systems and software process mismatches, lack of interdisciplinary teamwork, lack of respect by systems for other disciplines and blaming other groups for system engineering failures.

Exploitation Measures: some suggestions for exploiting agile productivity gains follow:

• Invite a customer/user representative to live with and participate as a member of the team.
• Truly embrace integrated product team or interdisciplinary team concepts with everyone placed on an equal footing. Have the customer/user representative participate as a team member.
• Earn engineering leadership positions instead of assuming them.
• Embrace an agile manifesto for engineering, not just for systems, hardware and software.
• Marry all engineering and manufacturing processes so that handover is synchronized and simplified.
• Develop a single agile-based engineering process instead of separate ones for individual disciplines (systems, software, hardware, specialty engineering, etc.).
• Focus process on meeting customer needs by developing working products early in process, not those of individual engineering disciplines.
• View requirements elicitation as an exploration rather than a specification process.
• Focus on developing working products throughout the development process instead of paperwork and its attendant bureaucracy.
• Embrace continuous integration practices as the unifying framework for all engineering.
• As a by-product of integration, provide the team a simple model of the complete system so that they can demonstrate what is built, ready for testing, in testing and ready to be used.
• Address needs for synchronization throughout the process by defining synch point to integrate the work-in-process at appropriate times.

8. Agile software maintenance - The jury is still out when it comes to agile software maintenance. Questions about the relative advantages and disadvantages of agile methods for this stage of development must be answered in light of the findings of a recent study that Reifer Consultants performed on software maintenance⁶. This study found that unlike during development where budgets and schedules were driven by the requirements, the resources allocated for software maintenance tasks were fixed. As such, as part of their work, software maintenance teams were asked to prioritize their work assignments (repairs, changes, updates, optimizations, etc.) so that they could get as much work as possible accomplished using the resources provided as part of their fixed budgets. Working off the accumulated change request and trouble report backlogs is therefore a natural part of their job.

The study also found that these same teams were tasked with developing the following four releases in parallel as part of their workload:

• Current release - the primary task for the maintenance team was developing a new release that updated the code to address the changes, repairs and optimizations approved by the customer.
• Fielded release - a priority task aimed at supporting fielded releases aimed at keeping the system operational by making necessary repairs and patches.
• Completed release - a secondary task was to configure and tailor the recently completed release for distribution to the field. This effort includes conducting more tests and coordinating with field sites.
• Requirements release - a secondary task aimed at developing the contents of the next release to be developed (i.e., may include prototyping as necessary to bound requirements).

The work associated with all four of these releases must be addressed by agile methods in order for it to perform well during software maintenance. Based on our studies, the composition and distribution of these work activities differ from that performed during software development. For example, much of this work is directed towards revalidating releases once changes have been made to them via regression testing. In addition, there may be field support and operational testing involved.

Root causes: misunderstandings about the nature of the work performed during maintenance and how maintenance activities/facilities are budgeted and adapted to handle enhancements, repairs and optimizations, both planned and emergency.

Issues: confusion over what work is performed on software development and maintenance projects, process/budget mismatches for maintenance, inadequate budgets leading to backlogs, revalidation of releases once changes have been made to them, product distribution management issues and poor product management discipline.

Exploitation Measures: some suggestions for exploiting agile productivity gains follow:

• Figure out how to adapt agile methods to the full scope of work associated with the four releases discussed above that needs to be performed during the software maintenance phase.
• Provide guidance on how agile techniques and practices can be used within the context of the processes used to develop the four releases identified earlier in the text. For example, sprints make sense for the current but not for the completed release.
• Develop a prioritization scheme for software maintenance work that focuses on keeping systems operational even as enhancement, repairs and optimizations are made in the field.
• Devise a suitable support (i.e., configuration, distribution, quality, supplier and risk management practices for all projects) infrastructure that can be used during software maintenance to keep different software releases up-to-date at different installations even when they are of different versions and heritages.
• Understand that maintenance facilities and customer support functions are budgeted separately. They are based on the total workload including the backlogs, not that of an individual project. Plan and staff to provide such support accordingly.
• Continue to embrace continuous integration and automated regression testing practices. Because testing in maintenance can consume as much as 75 percent of the effort when revalidating releases due to changes, these activities have the biggest impact.
• Maintain your process discipline because relaxing it during maintenance can lead to catastrophe especially when dealing with multiple releases, customers and product versions.

9. Agile risk management - Agile risk management practices need to be embraced especially as agile projects increase in scale and difficulty. While some techniques exist and are used for agile risk management like risk burn-down lists, few organizations in our database embrace them. Those that did were unsurprisingly within the defense and telecommunications domains.

Root Causes: risk management viewed as heavyweight process, team leaders not viewed as project managers (i.e., not tasked to manage risk), and poor project management discipline and practice (i.e., especially on agile projects of scale).

Issues: remove obstacles hindering delivery of quality products on-time and within budget, improve chances of meeting customer expectations and address inability to identify and mitigate risk.

Exploitation Measures: some suggestions for exploiting agile productivity gains follow:

• Put agile risk management practices to work to engage the team, management, customers and users in actions aimed at proactively identifying, quantifying, prioritizing (i.e., based on potential impact) and mitigating risks.
• Develop a “top 10” risk list, post it on the task boards and status it at the daily standup meetings and on task-boards.
• Prioritize the risks on the “top 10” list by potential technical and programmatic impact so that you know what to attack first. Use techniques like a risk burn-down risk to track mitigation.
• Engage the customer/user as part of the risk management process and have him/her help prioritize the risks based on your assessment of impacts. Use techniques like a risk council to reach consensus on the prioritization.
• Take risks when it makes sense and the consequences are not dire.
• Reward risk-taking, but do so when the rewards are worthwhile and you can exercise limits and controls on the potential consequences.

10. Agile contracting - Agile contracting practices need to be refined. Currently, contracting specialists and buyers do not know how to specify contract types, work scope, work statements, deliverables, milestones, payments, and terms and conditions for agile contracts/subcontracts. This leads to use of time and materials contracts for agile instead of those that provide incentives to reward organizations for superior performance.

Root Causes: agile is a relatively new way of doing business, few firms doing agile have involved buyers and few model contracts are available to work from.

Issues: lack of incentives and rewards for superior/inferior performance, few contractual controls and safeguards, and buyers avoid agile as they view it as uncontrollable.

Exploitation Measures:

• Develop a contractual template that software buyers can use for contracting/subcontracting for software developments using agile methods that identifies milestones and deliverables and contains contract language which they can use to reward (incentives, etc.) superior performance and provide remedies (cancellation clauses, etc.) for non-compliance issues.
• Develop a model agile contract/subcontract that provides a reasonable set of terms and conditions for agile acquisitions (including payment terms and milestones).
• Develop a template for deliverables that describes their form and format and provides for electronic rather than physical access to the work-in-process repositories at the supplier’s plant.
• Develop a legal template for permitting the customer/user representative to participate as a member of the development team under a limited liability umbrella clause.
• Develop training and certification courses in agile contracting/subcontracting for buyers and contract specialists working in firms embracing the concepts.

II. Major Findings

Because agile methods represents a new way of business for most organizations surveyed, the “take aways” discussed in the previous paragraph should be expected. Many of the issues highlighted like agile contracting and scaling continue to plague organizations that have made the transition. Based on everything we learned, the five major findings that we can glean relative to making the move to agile methods are the following:

1. While there is a definite business case that can be made for the use of agile methods, there are serious barriers to adoption that need to be overcome by newcomers to exploit use of the methodology.
2. Change management principles dominate as you move to agile methods. When making the transition, strategies need to developed, plans put into place, infrastructure revamped, staff trained, management educated and pilots run to figure out the best way to run agile projects within the context of your organization, its business goals, its processes and its past history. In addition, measure your progress and use metrics to help guide you on your path to success. One good way to avoid the traps that have snared others in the past is to bring in an agile coach and mentor as part of your team.
3. Agile methods may not work well for certain types of software development projects. As noted in the text of this white paper, other lightweight but non-agile methods (RUP, TSP, etc.) may work better for projects that are safety and security critical because of the certifications involved. In addition, contract issues make the use of agile methods very difficult to pursue in an acquisition environment because they represent roadblocks to getting the job done right. Again, a good agile expert can help you make determinations and findings relative to what methods to use, when, why and with what value to the customer/user.
4. Scaling of agile projects continues to be a problem because of organizational and process mismatches and the lack of project management discipline. When conflicts between groups dominate, little gets done. To avoid these issues, we recommend moving to a single engineering discipline that embraces affected groups, exploits true interdisciplinary team concepts, revamps processes with agile concepts in mind and takes full advantage of traditional project management principles like those advocated by the Project Management Institute in their Body of Knowledge (PMBOK)⁷. In this case, a good management expert who possesses agile, process improvement and project management skills, knowledge and abilities can be of assistance especially if he/she has experience within your industry and applications domain in which you operate.
5. From the competitive landscape, the move to agile methods often makes a lot of sense for three reasons. First, you can use the methodology to catch up with your competitors who have most likely already have made the commitment to use agile methods. Second, you can use agile methods as a means to attract talent. The reason for this is that high performers want to work in an environment that uses the latest and greatest techniques. Third and finally, you can use agile to show your customers that you are adopting techniques that makes their desires number one.

For those of  you interested in learning more about agile and its benefits and weaknesses, agile return-on-investment (ROI), the ten year trends and the details associated with our agile software productivity, cost and quality findings, we suggest you order the report entitled “Quantitative Analysis of Agile Methods” from either info@reifer.com or from here. These agile software productivity, cost and quality results are reported by domains in one of the following ten applications areas

• Automation
• Command and control
• Financial
• Defense
• Information technology
• Medical
• Mobile applications
• Software tools
• Telecommunications
• Web business   

References

¹ Reifer Consultants LLC, Quantitative Analysis of Agile Methods, No. 7, v1.3, 1 July 2013.

² Capers Jones, Comparing Software Methods and Results in a Side-by-side Format, Draft, Namcook, 25 June 2013.

³ Ahmad K. Shuja and Jochen Krebs, IBM Rational Unified Process Reference and Certification Guide: Solution Designer (RUP), IBM Press, 2008.

⁴ Watts S. Humphrey, TSP: Leading a Development Team, Addison-Wesley, 2005.

⁵ Henrik Kniberg, Lean from the Trenches: Managing Large-Scale Projects with Kanban, Pragmatic Bookshelf, 2011.

⁶ Donald J. Reifer, Software Maintenance Success Recipes, CRC Press, 2011.

⁷ See this here.

Acknowledgements

The author acknowledges the useful comments offered by reviewers of this document.
It should also be noted that many of the experts reviewing the document have collected data on agile projects which confirms many of the findings and conclusions offered in this white paper.

About the Author

Reifer Consultants LLC is an Arizona Limited Liability Company that specializes in the area of software engineering economics, metrics and measurement, benchmarking and empirical analysis. Since its founding in 1980, Reifer Consultants has provided its clients with a full range of software services including preparing software business strategies and cases, developing independent estimates, inserting metrics and measurement programs, developing estimating models, conducting competitive analyses and performing independent risk assessments. The firm has also over a decade of experience coaching and advising clients on how to take full advantage of agile methods while avoiding their many disadvantages. They have helped over a dozen Fortune 500 companies make an orderly transition to agile methods. Contact Information: Reifer Consultants LLC, 14820 N. Dragons Breath Lane, Prescott, AZ 86305, Phone: 928-237-9060, Email: donald.reifer@gmail.com.

Warnings

This publication was developed to provide authoritative information with regard to its subject matter. Reifer Consultants disclaims any warranties or conditions with regard to the specific content, express or implied, including warranties of merchantability and fitness for a specific use. In addition, Reifer Consultants does not assume any legal liability for the accuracy, completeness and/or usefulness of any information contained within this publication. Finally, any reference to a commercial product, process, or service within this document does not imply or constitute an endorsement by Reifer Consultants.

All trade names, trademarks, or registered trademarks are trade names, trademarks or registered trademarks of their respective owners.