The 2014 CAST Research on Application Software Health (CRASH) report states that enterprise software built using a mixture of agile and waterfall methods will result in more robust and secure applications than those built using either agile or waterfall methods alone.
Data from CAST’s Appmarq benchmarking repository was analyzed to discover global trends in the structural quality of business application software. The report explores the impact of factors such as development method, CMMI maturity level, outsourcing, and other practices on software quality characteristics that are based upon good architectural and coding practices.
You can download a copy of the executive summary of the 2014 CRASH report (registration required) from the CAST website.
InfoQ interviewed Bill Curtis, Senior Vice President and Chief Scientist at CAST, about the research done by CAST, structural quality factors, and mixing agile and waterfall methods.
InfoQ: For readers that do not know CAST, can you describe what CAST is?
Bill: CAST (www.castsoftware.com) has developed a technology called the Application Intelligence Platform (AIP) for analyzing the structural quality of software in large multi-layer, multi-language business systems. Structural quality consists of the non-functional, internal attributes of a system such as Robustness, Performance Efficiency, Security, Changeability, etc. CAST AIP detects over 1200 violations of good architectural and coding practice at both the system and code unit levels in 28 different languages such as Java, .NET, C, C++, COBOL, PHP, and ABAP. The information and measures resulting from the analysis provides developers with the location and severity of weaknesses, as well as providing executives with visibility into the risks and costs of their application portfolio.
InfoQ: Why is structural quality important?
Bill: Structural quality represents the engineering of an application—not whether it computes an answer correctly, but whether the architecture and code is constructed in a manner that avoids crashes, unauthorized access, data corruption, inability to scale, excessive complexity, and similar problems. Damages when these problems strike business critical applications can run into the millions, and in the worst cases they have made software quality a boardroom issue.
As Diomedis Spinellis states in his excellent book Code Quality, structural problems are very difficult to detect through traditional testing, and the majority of test cases are designed to check functional correctness. The most damaging structural defects frequently result from interactions among components in different architectural layers where assumptions about the interactions prove incorrect. These architecturally complex defects are rarely detected with unit level static analysis. Research has shown that although they rarely account for more than 10% of the structural problems in a system, they consume over half of the rework effort because of the number of files involved.
InfoQ: Which of the structural quality factors pose the biggest problems in organizations?
Bill: While I can make a case for several structural quality factors, Security and Reliability are getting the most attention since breaches and critical outages seem to be reported daily. Embarrassingly, we have known about many of the most frequently exploited Security weaknesses such as SQL injection for years. With current advances in detection capabilities, why haven’t these weaknesses been removed? For many applications, especially if they are customer facing, Reliability is critical for avoiding outages during peak hours and recovering quickly when they occur.
InfoQ: Why did CAST do this research on application software health?
Bill: Many of our customers submit the results of their application system analyses to our Appmarq Repository so that they can benchmark their structural quality scores. The anonymized data provide a rich source for research on structural quality and the factors that affect it. In addition to the vast scientific value of these data, we want to show customers how they can use their internal data to answer questions that drive improvement. We are trying to get IT to understand that analyzing the structural quality of their products is just as important as it would be if they were in manufacturing. Continual improvement must involve the product as much as it has involved the process and method.
InfoQ: How was the research done?
Bill: We drew a sample from Appmarq of all applications containing at least 10,000 lines of code (LOC). The 2014 sample consisted of 1,316 applications from 212 organizations in 12 industry sectors primarily in North America, Europe, and Asia. The median size was 128,000 LOC, but the mean was 470,000 LOC since 143 applications were over 1 million LOC. Many of the application owners provided descriptive information on location, development method, CMMI Level, and other factors. In some cases it is difficult to compare analyses across languages and technologies because of differences in the types of structural violations that can occur. Just over 40% of the applications were written in Java-EE, and we had enough descriptive information on them to support the analyses in the CRASH Report.
InfoQ: One of the conclusions is that "Across all [structural quality characteristics], a mix of Agile and Waterfall methods produced higher scores than either Agile or Waterfall methods alone". Can you elaborate?
Bill: Consistently across all the structural quality characteristics—Robustness, Performance Efficiency, Security, Changeability, etc.—the mix of Agile and Waterfall methods had significantly higher scores. In fact, for Robustness and Changeability ¾ of the scores for the hybrid methods were higher than the median for Agile or Waterfall methods used alone. In addition, there was less variance in the structural quality scores on the hybrid methods. Remember, these are primarily large business critical applications, so late corrections to the architecture are very expensive and often constrained. Believing that a satisfactory architecture for such systems will emerge across numerous Agile sprints can be risky. However, not evaluating architectural decisions until system testing in Waterfall projects is also risky. Good structural quality was most often observed when the early architectural analysis characteristic of Waterfall methods was combined with continual testing of the emerging software in Agile sprints. We did not see significant differences between Agile and Waterfall methods in structural quality, it was really their combination into hybrid methods that made a difference.
InfoQ: I also noticed some findings on sourcing and process maturity?
Bill: Consistent with previous CRASH Reports, we did not find any differences between applications that had been developed in-house or outsourced. The major factors affecting structural quality appear to be something other than the sourcing of development.
We did find that the structural quality of software built in CMMI Level 1 organizations was significantly lower than that developed in more mature environments. When developers are ensnared in a Level 1 death march, they are working overtime against unachievable schedules and consequently produce code with many more structural weaknesses that they do not have time to detect or refactor. Getting control of commitments and baselines allows developers to work at a more sustainable pace and produce higher code quality. I have even heard developers on Agile projects complain about adding numerous stories during a sprint under pressure from the business thus overcommitting the team’s work. Sprints are best served frozen, not thawed.
The objective of CMMI Level 2 is to establish control over commitments and baselines on each project or application so that developers are protected from perpetual firefighting and chaos, enabling them to work at an orderly and professional pace. Embracing change does not imply embracing chaos. We did not see differences between applications in CMMI Level 2 and Level 3 organizations because the transition between Levels 2 and 3 is primarily an organizational initiative to establish common practices that can be adapted to different projects in order to develop consistent understanding, organizational learning, and an economy of scale.
InfoQ: Is there any advice that you want to give to enterprises that have implemented or are implementing agile software development?
Bill: These results confirm with hard data what many Agile thought leaders such as Alistair Cockburn, Scott Ambler, Dean Leffingwell, and Todd Little said for years. Agile projects should adjust their practices to the size, complexity, and uncertainty of the application challenges they face. The larger more complex the challenge, the more removing some of the architectural uncertainty early helps avoid rework and technical limitations in later sprints. Agile projects should also be evaluating system level structural quality after several of the builds in each sprint in order to detect problems between different subsystems or layers of the application.
Some uncertainty is functional while some is structural. Agile Methods have focused on managing the functional uncertainty arising from the changing needs of customers. Structural uncertainty is often relegated to ‘emergent design’. However, refactoring architectures in large systems can be excessively expensive and slow the pace of delivering valuable software. So look at development as a collection of practices you can adjust and balance to manage the levels of structural and functional uncertainty in each system. Be agile with your methods.