The Continuous Delivery Maturity Model

The principles and methods of Continuous Delivery are rapidly gaining recognition as a successful strategy for true business agility. For many organizations the question is no longer “why?”, but rather “how?” How do you start with Continuous Delivery, and how do you transform your organization to ensure sustainable results. This Maturity Model aims to give structure and understanding to some of the key aspects you need to consider when adopting Continuous Delivery in your organization.

Why a maturity model?

Continuous Delivery is all about seeing the big picture, to consider all aspects that affect the ability to develop and release your software. For any non-trivial business of reasonable size this will unfortunately include quite a lot of steps and activities. The end-to-end process of developing and releasing software is often long and cumbersome, it involves many people, departments and obstacles which can make the effort needed to implement Continuous Delivery seem overwhelming. Where do we start? Do we have to do everything, what parts can we leave out? Where do we get the biggest and quickest effect? These are questions that inevitably will come up when you start looking at implementing Continuous Delivery.

This is why we created the Continuous Delivery Maturity Model, to give structure and understanding to the implementation of Continuous Delivery and its core components. Inspiration for this model comes mainly from our combined experiences in several continuous delivery implementation projects, and also from selected books, papers and blog-posts on the subject where Jez Humble & David Farley’s groundbreaking book Continuous Delivery and Eric Minick & Jeffrey Fredricks terrific whitepaper Enterprise Continuous Delivery Model are two that deserve special recognition. With this model we aim to be broader, to extend the concept beyond automation and spotlight all the key aspects you need to consider for a successful Continuous Delivery implementation across the entire organization.

A structured approach

The journey that started with the Agile movement a decade ago is finally getting a strong foothold in the industry. Business leaders now have begun to embrace the fact that there is a new way of thinking about software development. A paradigm shift, if you like, that will inevitably segment the industry into those who struggle to keep up with competition, and those who have the ability to stay agile and ahead of competition by having an IT organization that responds quickly and efficiently, and serves as a true business enabler. IT can once again start pushing innovation instead of restraining it by expensive, slow, unpredictable and outdated processes. There are many ways to enter this new era and here we will describe a structured approach to attaining the best results. While agile methodologies often are described to best grow from inside the organization we have found that this approach also has limitations. Some parts of the organization are not mature enough to adapt and consequently inhibit development, creating organizational boundaries that can be very hard to break down. The best way to include the whole organization in the change is to establish a solid platform with some important prerequisites that will enable the organization to evolve in the right direction. This platform includes adopting specific tools, principles, methods and practices that we have organized into five key categories, Culture & Organization, Design & Architecture, Build & Deploy, Test & Versification and Information & Reporting. Structuring Continuous Delivery implementation into these categories that follows a natural maturity progression will give you a solid base for a fast transformation with sustainable results.

The purpose of the maturity model is to highlight these five essential categories, and to give you an understanding of how mature your company is. Your assessment will give you a good base when planning the implementation of Continuous Delivery and help you identify initial actions that will give you the best and quickest effect from your efforts. The model will indicate which practices are essential, which should be considered advanced or expert and what is required to move from one level to the next.

Five levels

The model defines five maturity levels: base, beginner, intermediate, advanced and expert. The model is calibrated with a typical industry average around the base level where we see most organizations are at today. Some organizations have beginner maturity in some categories and below base (outside the model) in others but on average base-level is the typical starting point for many organizations. The intermediate level is what we consider a mature level of continuous delivery practices where you benefit from the larger effects. The advanced level includes practices that will bring substantial value and effect to a higher effort. The last maturity level, expert, includes practices that will be very valuable for some organizations that want or need to specialize in certain practices.

The levels are not strict and mandatory stages that needs to be passed in sequence, but rather should serve as a base for evaluation and planning. It is however important to try to keep the overall maturity level fairly even and to keep in mind that big changes may cause skepticism and reluctance in the organization, so an incremental approach to moving through the levels is recommended.

Five categories

The model also defines five categories that represent the key aspects to consider when implementing Continuous Delivery. Each category has it’s own maturity progression but typically an organization will gradually mature over several categories rather than just one or two since they are connected and will affect each other to a certain extent.

Culture & Organization

The organization and it’s culture are probably the most important aspects to consider when aiming to create a sustainable Continuous Delivery environment that takes advantage of all the resulting effects.

Base

A typical organization will have, at base level, started to prioritize work in backlogs, have some process defined which is rudimentarily documented and developers are practicing frequent commits into version control.

Beginner

Moving to beginner level, teams stabilize over projects and the organization has typically begun to remove boundaries by including test with development. Multiple backlogs are naturally consolidated into one per team and basic agile methods are adopted which gives stronger teams that share the pain when bad things happen.

Intermediate

At the intermediate level you will achieve more extended team collaboration when e.g. DBA, CM and Operations are beginning to be a part of the team or at least frequently consulted by the team. Multiple processes are consolidated and all changes, bugs, new features, emergency fixes, etc, follow the same path to production. Decisions are decentralized to the team and component ownership is defined which gives teams the ability to build in quality and to plan for sustainable product and process improvements.

Advanced

At the advanced level, the team will have the competence and confidence it needs to be responsible for changes all the way to production. Continuous improvement mechanisms are in place and e.g. a dedicated tools team is set up to serve other teams by improving tools and automation. At this level, releases of functionality can be disconnected from the actual deployment, which gives the projects a somewhat different role. A project can focus on producing requirements for one or multiple teams and when all or enough of those have been verified and deployed to production the project can plan and organize the actual release to users separately. This separation of concerns will optimize the projects flexibility of packaging and releasing functionality and at the same time give the development teams better flow and speed since they don’t have to stack up changes and wait for coordinated project releases.

Expert

At expert level some organizations choose to make a bigger effort and form complete cross functional teams that can be completely autonomous. With extremely short cycle time and a mature delivery pipeline, such organizations have the confidence to adopt a strict roll-forward only strategy to production failures.

Design & Architecture

The design and architecture of your products and services will have an essential impact on your ability to adopt continuous delivery. If a system is built with continuous delivery principles and a rapid release mind set from the beginning, the journey will be much smoother. However, an upfront complete redesign of the entire system is not an attractive option for most organizations, which is why we have included this category in the maturity model.

Base

A typical organization will have one or more legacy systems of monolithic nature in terms of development, build and release. Many organizations at the base maturity level will have a diversified technology stack but have started to consolidate the choice of technology and platform, this is important to get best value from the effort spent on automation.

Beginner

At beginner level, the monolithic structure of the system is addressed by splitting the system into modules. Modules give a better structure for development, build and deployment but are typically not individually releasable like components. Doing this will also naturally drive an API managed approach to describe internal dependencies and also influence applying a structured approach to manage 3rd party libraries. At this level the importance of applying version control to database changes will also reveal itself.

Intermediate

Moving to intermediate level will result in a solid architectural base for continuous delivery when adopting branch by abstraction  and other techniques for feature hiding for the purpose of minimizing repository branching to enable true continuous integration. At this level the work with modularization will evolve into identifying and breaking out modules into components that are self-contained and separately deployed. At this stage it will also be natural to start migrating scattered and ad-hoc managed application and runtime configuration into version control and treat it as part of the application just like any other code.

Advanced

At the advanced level you will have split the entire system into self contained components and adopted a strict api-based approach to inter-communication so that each component can be deployed and released individually. With a mature component based architecture, where every component is a self-contained releasable unit with business value, you can achieve small and frequent releases and extremely short release cycles. At this level it is easy for the business to experiment with rapid release of new features, monitor and verify expected business results; therefore techniques to push and graph business metrics from the application becomes important and natural part of the overall design and architecture.

Expert

At expert level, some organizations will evolve the component based architecture further and value the perfection of reducing as much shared infrastructure as possible by also treating infrastructure as code and tie it to application components. The result is a system that is totally reproducible from source control, from the O/S and all the way up to application. Doing this enables you to reduce a lot of complexity and cost in other tools and techniques for e.g. disaster recovery that serves to ensure that the production environment is reproducible. Instead of having a separate process, disaster recovery is simply done by pushing out the last release from the pipeline like any other release. This together with virtualization gives extreme flexibility in setting up test and production environments with minimum manual effort.

Build & Deploy

Build and deployment is of course core to Continuous Delivery and this is where a lot of tools and automation come into the pipeline; this is what is most is commonly perceived when Continuous Delivery is discussed. At first glance a typical mature delivery pipeline can be very overwhelming; depending on how mature the current build and deployment process is in the organization, the delivery pipeline can be more or less complex. In this category we will describe a logical maturity progression to give structure and understanding to the different parts and levels it includes.

Base

At a base level you will have a code base that is version controlled and scripted builds are run regularly on a dedicated build server. The deployment process is manual or semi-manual with some parts scripted and rudimentarily documented in some way.

Beginner>

Beginner level introduces frequent polling builds for faster feedback and build artifacts are archived for easier dependency management. Tagging and versioning of builds is structured but manual and the deployment process is gradually beginning to be more standardized with documentation, scripts and tools.

Intermediate

At intermediate level, builds are typically triggered from the source control system on each commit, tying a specific commit to a specific build. Tagging and versioning of builds is automated and the deployment process is standardized over all environments. Built artifacts or release packages are built only once and are designed to be able to be deployed in any environment. The standardized deployment process will also include a base for automated database deploys (migrations) of the bulk of database changes, and scripted runtime configuration changes. A basic delivery pipeline is in place covering all the stages from source control to production.

Advanced

At this stage it might also become necessary to scale out the build to multiple machines for parallel processing and for specific target environments. Techniques for zero downtime deploys can be important to include in the automated process to gain better flexibility and to reduce risk and cost when releasing. At this level you might also explore techniques to automate the trailing part of more complex database changes and database migrations to completely avoid manual routines for database updates.

Expert

Expert practices will include zero touch continuous deployment to production where every commit can potentially make it all the way to production automatically. Another expert practice is taking the infrastructure as code and virtualization even further and making the build process produce not only deployment artifacts but complete virtual machines which are promoted down the pipeline all the way to production replacing the existing ones.

Test & Verification

Testing is without doubt very important for any software development operation and is an absolutely crucial part of a successful implementation of Continuous Delivery. Similar to Build & Deploy, maturity in this category will involve tools and automation. However, it is also important to constantly increase the test-coverage of the application to build up the confidence in speed with frequent releases. Usually test involves verifying expected functionality according to requirements in different ways but we also want to emphasize the importance of verifying the expected business value of released features.

Base

At the base stage in the maturity model a development team or organization will typically practice unit-testing and have one or more dedicated test environments separate from local development machines. This system and integration level testing is typically done by a separate department that conducts long and cumbersome test periods after development "code freeze".

Beginner

When moving to beginner level you will naturally start to investigate ways of gradually automating the existing manual integration testing for faster feedback and more comprehensive regression tests. For accurate testing the component should be deployed and tested in a production like environment with all necessary dependencies.

Intermediate

Beginner level lets you broaden the scope of your automatic tests to functional testing which includes bigger parts of the component than unit tests but will have “mocked” implementations of it’s external dependencies, e.g. database or other backend services. These tests are especially valuable when working in a highly component based architecture or when good complete integration tests are difficult to implement or too slow to run frequently. At this level you will most likely start to look at gradually automating parts of the acceptance testing. While integration tests are component specific, acceptance tests typically span over several components and across multiple systems.

Advanced

Advanced practices include fully automatic acceptance tests and maybe also generating structured acceptance criteria directly from requirements with e.g. specification by example and domains specific languages. This means no manual testing or verification is needed to pass acceptance but typically the process will still include some exploratory testing that feeds back into automated tests to constantly improve the test coverage and quality. If you correlate test coverage with change traceability you can start practicing risk based testing for better value of manual exploratory testing. At the advanced level some organizations might also start looking at automating performance tests and security scans.

Expert

It might seem strange to state that verifying expected business result is an expert practice but this is actually something that is very rarely done as a natural part of the development and release process today. Verifying expected business value of changes becomes more natural when the organization, culture and tooling has reached a certain maturity level and feedback of relevant business metrics is fast and accessible. As an example the implementation of a new feature must also include a way to verify the expected business result by making sure the relevant metrics can be pulled or pushed from the application. The definition of done must also be extended from release to sometime later when business has analyzed the effects of the released feature or change..

Information & Reporting

Any business process includes a specific information set which can be reported on; the process of developing and releasing software is no exception, and the set of information that is handled in this particular area would typically include concepts like component, requirement, version, developer, release, environment etc. In this category we want to show the importance of handling this information correctly when adopting Continuous Delivery. Information must e.g. be concise, relevant and accessible at the right time to the right persons in order to obtain the full speed and flexibility possible with Continuous Delivery. Apart from information directly used to fulfill business requirements by developing and releasing features, it is also important to have access to information needed to measure the process itself and continuously improve it.

Base

At the base level in this category it is important to establish some baseline metric for the current process, so you can start to measure and track. At this level reporting is typically done manually and on-demand by individuals. Interesting metrics can e.g. be cycle-time, delivery time, number of releases, number of emergency fixes, number of incidents, number of features per release, bugs found during integration test etc.

Beginner

At beginner level, you start to measure the process and track the metrics for a better understanding of where improvement is needed and if the expected results from improvements are obtained. Reporting at this stage would typically include static analysis of code and quality reports which might be scheduled so that the latest reports are always accessible to facilitate decisions on quality and where improvements are needed.

Intermediate

Moving to intermediate the level of automation requires you to establish a common information model that standardizes the meaning of concepts and how they are connected. This model will typically give answers to questions like; what is a component?; how are requirements related to changes and components? When this model is established not only can you automate build, test and deployment even further but you can also build in traceability and information transparency to the pipeline and e.g. automatically generate information like release notes and test plans. Automatic reporting and feedback on events is implemented and at this level it will also become natural to store historical reports connected to e.g. builds or other events. This gives management crucial information to make good decisions on how to adjust the process and optimize for e.g. flow and capacity.

Advanced

At advanced level when you have started to practice frequent releases and the tools, products and services are mature enough to e.g. push business metrics, a natural progression is to set up a graphing service where people can get specific real time information that is relevant to their specific need. At this level real time graphs and other reports will typically also include trends over time. As a complement to static code analysis and unit tests coverage reports you might also at this level start looking at dynamic test coverage and profiling information from production like runtime environments when e.g. running automatic integrations tests.

Expert

Moving to expert level in this category typically includes improving the real time information service to provide dynamic self-service useful information and customized dashboards. As a result of this you can also start cross referencing and correlating reports and metrics across different organizational boundaries,. This information lets you broaden the perspective for continuous improvement and more easy verify expected business results from changes.

Jump start the journey

Every company is unique and has its own specific challenges when it comes to changing the way things work, like implementing Continuous Delivery. This maturity model will give you a starting point and a base for planning the transformation of the company towards Continuous Delivery. After evaluating your organization according to the model you need to set the goals and identify which practices will give your organization the best outcomes. If there are practices you do not want to adopt you need to analyse the consequences of excluding them. It is also important to decide on an implementation strategy, you can e.g. start small using slack in the existing process to improve one thing at a time. However, from our experience you will have a better chance of a successful implementation if you jump start the journey with a dedicated project with a clear mandate and aggressive goals on e.g. reducing cycle time. A typical Continuous Delivery implementation project will need skills and competence according to the categories and practices in the maturity model in order to implement a solid platform of tools, methods and practices that the organization can continue to grow and improve on.

(Click on the image to enlarge it)

About the authors

Andreas Rehn is an Enterprise Architect and a strong advocate for Continuous Delivery, DevOps, Agile and Lean methods in systems development. With extensive experience in many disciplines of software development and a solid understanding of process, information and management theories and practises; he is dedicated to help customers implement Continuous Delivery and transform their business by adopting new methods for efficient and modern software development.

Tobias Palmborg, Believes that Continuous Delivery describes the vision that scrum, XP and the agile manifesto once set out to be. Continuous Delivery is not just about automating the release pipeline but how to get your whole change flow, from grain to bread ,in a state of the art shape. Former Head of Development at one of europes largest online gaming company. Tobias is currently implementing Continuous Delivery projects at several customers.

Patrik Boström is one of the founders of Diabol. Senior developer and architect with experience in operations of large system. Strong believer that Continuous Delivery and DevOps is the natural step in the evolution of Agile and Lean movement. Wants to change the way we look at systems development today, moving it to the next level where we focus more time on developing features than doing manually repetitive tasks. Where we visualize and understand the path from idea to where it is released and brings business value.