Jez Humble has stated that current software delivery practices are not optimised to create valuable software, and three issues must be addressed in order to enable innovation. First, the traditional project model is unsuitable. Second, the entire organisational value stream must be addressed. Third, the problems are rooted in process and culture, not organisational structure or tooling.
Humble, a co-author of Continuous Delivery and Lean Enterprise, began the talk at the Jfokus conference in Stockholm by stating that contemporary software delivery is typically based around the project model. This model was originally developed by the US military for building things at scale, such as missiles and aeroplanes.
Humble argued that this approach is valid for things that do not change greatly once they have been built, where significant information is not discovered in the process of building, and where the building process must be finished before the thing can be used. However, none of this is true for modern software.
Over the last fifteen years the software development ‘methodology wars’ have been fought. Popular methodologies have included the waterfall model, the V model, and the lightweight methodology from which agile methodologies emerged.
Humble stated that although the introduction of the agile manifesto drove change, many organisations now implement a ‘water-scrum-fall’ methodology in which only the development phase of a project is performed in an iterative manner. The ‘fuzzy front end’ of study, design and planning is still largely a single process, and so is the ‘last mile’ of the quality assurance, release and operation phases.
The absence of iteration in the study, design and planning phases may prevent what is generally referred to as negative ‘scope creep’, however, Humble argued that this also limits innovation. Additionally, the ‘water-scrum-fall’ methodology does not enable software to be released early to end-users for validation, and does not result in reduced lead times.
The first principle of the agile manifesto states that “our highest priority is to satisfy the customer through early and continuous delivery of valuable software”, and Humble suggested that the combination of processes such as the ‘Lean Startup’ methodology and a ‘DevOps’ culture enable rapid iteration and a process optimised for fast learning and the identification of product/market fit.
Eric Reis’ The Lean Startup methodology introduced the process of ‘build, measure, learn’ applied to the creation of valuable products and services, and also to the validation of the existence of an associated market (the product/market fit). Humble stated that this process is essentially the scientific method applied to the task of identifying a valid business model, with a minimum viable product (MVP) being created as an experiment to determine the value of the product of service to a well-defined audience.
An MVP is created from an initial hypothesis about a potential target audience or market, with a clear set of business metrics defined that would indicate a positive result. Humble stated that the 80/20 rule to development effort should be applied here, and only functionality implemented that meets the small well-defined set of requirements around the hypothesis.
The MVP should not be concerned with implementing a highly scalable solution, and edge-cases may be ignored. The primary goal is to build features quickly, deploy the product and gather data in the form of business metric from actual users.
When designing experiments Humble suggested following the work of Jeff Gothelf and Josh Seiden in ‘Lean UX’, where the hypothesis should take the form:
We believe that
[building this feature]
[for these people]
will achieve [this outcome].
We will know we are successful when we see [this signal from the market].
Humble stated that A/B tests are the gold standard of demonstrating causal relationships between features delivered and associated impact for end-users. Many companies perform software releases with hundreds of changes without a mechanism to determine which feature resulted in positive or negative impact.
In 2009 Ronny Kohavi published a Microsoft Thinkweek paper “Online Experimentation at Microsoft”, which stated that only one in three well-designed and executed experiments at Microsoft were successful at improving a targeted key metric. Humble argued that as effectively two-thirds of a project’s software development time may be wasted, early use of experimental techniques such as A/B testing are essential in preventing work on features that are not delivering value to users.
Designing A/B test experiments is not trivial, and factors such as establishing a large enough pool of users and watching for interaction between concurrently running experiments must be careful controlled. Humble stated that in his opinion this is currently the biggest skills gap in the software delivery industry, as there is no body of knowledge for doing this effectively nor is it not taught effectively in MBA or other university courses.
Small pockets of organisations with these skills are emerging, and one example of a company that utilises A/B testing effectively is Etsy. Etsy have released details of their experimental approach via their codeascraft blog and also presentations at QCon, such as “Etsy’s Product Development with Continuous Experimentation”.
Humble also discussed the creation of a statistically valid model for IT performance used in the Puppetlabs 2014 State of DevOps survey, which included ‘throughput’ measured in lead time for changes and change frequency, and ‘stability’, which was the time taken to restore service after an incident. The survey results showed that external change requests slow throughput, but do not increase stability, and that predictors of high performance within an organisation included proactive monitoring, organisational culture and a win-win relationship between development and operations.
The final topic discussed was the importance of dealing effectively with failure in an large organisation. Humble suggested that no one has perfect knowledge or is capable of fully understanding the side effects in a complex system such as modern software applications and the organisations they model. Failure in a complex system should always lead to enquiry, not blame, and the output of an incident postmortem should aim to enable more information and provide additional less risky actions to allow the problem to be resolved.
Humble concluded the presentation by stating that a high trust culture is correlated with innovation, and suggested that everyone within the organisation must be empowered to experiment, learn and innovate. This combined with passion and appropriate processes will enable the delivery of more valuable software.
The slides for the talk can be found on the Jfokus website, and the topics contained within this article are discussed in further depth in Jez Humble’s new book, Lean Enterprise.
