August Lilleaas, the host of Norway’s utviklingslandet.no software development podcast, recently wrote about the correlation between organization complexity and software quality. Rapid Software Testing Methodology creator James Bach has also recently written a piece titled "Assess Quality. Don’t Measure It," about how we should remain cautious in interpreting subjective measures of quality and not lose sight of user satisfaction.
A recent InfoQ interview with the authors of Team Topologies also shared insights into how adapting organizational structure can improve the health of software products.
Writing about a 2008 Microsoft paper titled The Influence of Organizational Structure On Software Quality: An Empirical Case Study, Lilleaas described a highly accurate model which was able to predict the quality of a software product, based on the distances in the organizational hierarchy between collaborating individuals.
Summarising the original study, Lilleaas explained that a model accounting for "organizational complexity" had been demonstrated to be highly effective in predicting software quality. This had been in contrast to a number of more traditional models which utilised specific types of static analysis and other software related metrics.
Lilleaas described Microsoft’s model which utilised several factors to predict software quality based on hierarchy and communication paths between collaborators within an organisation. Lilleaas listed several of the factors used in this model which reliably predicted the likelihood of "failure-proneness" in modular components of Windows Vista:
- The "number of developers working on the module" used within one or more software products
- The number of ex-developers that previously worked on that module
- The fraction of the organization that has worked on a module
- The degree of separation in the organisation between collaborators who impact a module, eg. "the distance in the organization between the developer and the decision-maker"
Given the model’s ability to predict software quality issues with a high degree of accuracy, Lilleaas conjectured that:
The distance to decision-makers and the number of developers working on a project is clearly and unambiguously the issue that is the best predictor of future problems with a code base.
In a recent Q&A with InfoQ, Matthew Skelton and Manuel Pais, the authors of Team Topologies, spoke of contemporary patterns for effectively structuring collaborative teams within organisations. Their book presents patterns which reduce the negative impact of organizational complexity on software architectures. Skelton and Pais explained the software impact which can arise from organizational change and why it should matter to leaders:
Organization design is effectively a constraint on the "solution search space" and so for a certain organization design, some solutions will never be discovered. That’s a strategic C-level concern.
Responding to a question on the nature of the Reverse Conway Maneuver, Skelton and Pais discussed how intentional structuring of an organisation to reflect desirable communication lines can organically influence a reduction in the complexity of software and architecture. Although being clear that this is not a panacea, they shared that it will nonetheless have an impact on improving the overall quality of your products:
The Reverse Conway manoeuvre does not guarantee effective software delivery, but it does result in more time and effort spent on meeting business goals rather than pushing against the communication mismatches between organization and software.
Focusing on the utility of quantitative metrics for quality, Bach discussed that whilst valuable, most quality metrics are inherently subjective when considered in relation to the product as a whole. Bach explains that "measurement is about assigning numbers to things." He asserts that a true appreciation of a product’s quality cannot be understood from a quantifiable metric alone. Bach wrote:
We want to understand the status of our products, but while many things can be measured, the quality of a product is not one of those things. At least, it can’t be measured in any sense that covers all of what customers, users, and business people are talking about when they wonder if the "quality of a product is good."
Bach warned against a "ritualistic use of numbers to cast a scientific aura around an otherwise irrational management process." He wrote of measuring software quality:
People keep trying to count things that don’t make any sense to count, such as test cases (since a test case is essentially a file that has something in it related to performing testing, that would be like judging a business by counting all the briefcases, drawers, cabinets, and envelopes in the building… "you have 42,564 containers of business stuff! Good businessing!")
Rather than discounting the value of such proxy metrics, Bach’s underlying lesson is that they should be interpreted with care as part of a continuous process of quality assessment focused on user happiness. He warned that it is very easy to play a "game of goal displacement" using "surrogate measures" which lose their end-user focus. He wrote:
People respond to the challenge of measuring quality simply by redefining it from something customers understand into something limited and legalistic. They define requirements in rigorous contractual terms. They make a set of "acceptance tests" that are asserted to prove that "the product works" if none of the tests "fail."
Bach ends with a reminder not to lose sight of the overarching measure of product quality, which he describes as satisfying your customers:
If you can get your customers to agree to play that game (to accept the displaced and surrogate goals and let go of the desire to feel happy with your product), and if you choose not to care whether they are actually happy with your product, maybe you will feel happy. However, the history of technology is littered with abandoned products and withered companies that ignored the elusive but persistent reality of quality. I, personally, don’t want to get rich while upsetting customers. Are you with me?
