Key Metrics to Track and Drive Your Agile Devops Maturity

We talk with many different organisations, all of whom are on some form of Agile transformational journey.  

Some are very early in their Agile transformation – just starting to implement an Agile way of working across their delivery teams, and are at the early stages of implementing the DevOps toolsets to underpin those processes.  Much of their software delivery may still be ‘waterfall’ in nature and they may not have yet achieved an effective CI/CD (continuous integration/continuous deployment) methodology. 

At the other end of the spectrum are those advanced or ‘Mature’ Agile practitioners who tend to be many months into their Agile transformation, with well established Agile methodologies (often at scale); an effective set of DevOps tooling; Delivery and Operations teams aligned to deliver in an Agile way; and business stakeholders who understand Agile principles.  As a result, these ‘mature’ Agile businesses are highly proficient at delivering quality software dependably, early and often. 

This article considers sensible metrics for these different types of organisations - from  ‘Early-stage’, to ‘Intermediate’ and ‘Advanced’ level Agile DevOps practitioners - so organisations can track their progress over time and self-improve around sensible metrics suited to the maturity of their Agile DevOps capability. 

It stands to reason that effective measurement of the end-to-end Agile delivery process is critical in order to track and drive improvement over time – and ultimately to deliver the core agile goal of “the early and continuous delivery of valuable software”.  As such, there are many agile, delivery and engineering metrics that can be used.  But how do you choose sensible metrics for your organisation’s level of Agile maturity?

This article discusses our recommended metrics for mature, intermediate, and early-stage Agile delivery practitioners.  It considers these metrics in a hierarchical way – with suitable metrics for technology leadership, delivery & engineering management, and crucially the teams themselves.  

Using metrics to understand the health of your delivery capability and your DevOps maturity

Agile software delivery is a complex process that can hide significant inefficiencies and bottlenecks.  

Fortunately the process is easily measureable as there is a rich digital footprint in the tool-sets used across the process – from pre-development; development; integration & deployment; and out into live software management. 

However surfacing data from these myriad data sources (toolsets) and synthesising meaningful metrics that compare ‘apples with apples’ across complex Agile delivery environments is very tricky.

Hence until recently, software delivery metrics have been much discussed but little used, until the arrival of Value Stream Management and BI solutions that enable the surfacing of accurate end-to-end software delivery metrics for the first time.  

As a result the field is moving centre stage, with Gartner and Forrester (to name but two) starting to advocate the importance of metrics and analytics in effective DevOps Value Stream Management (See Gartner Sept 2020 Market Guide ‘DevOps Value Stream Management Platforms’).

Metrics for early-stage Agile DevOps practitioners

Here we’ve selected a few simple metrics that in our experience are ideal for organisations at the early stage of an Agile DevOps transformation. The metrics focus on simple measures that underpin the fundamentals of increased agility and are relatively easy to understand and act upon.

Figure 1 - Early-stage Agile Devops Maturity metrics

Cycle Time is an ideal delivery metric for early stage practitioners.  It simply measures the time taken to develop an increment of software.  Unlike the more comprehensive measure of Lead Time (which measures the length of the entire end-to-end delivery process), Cycle Time is easier to measure as it looks only at the time taken (within a scrum team) to take a ticket from the backlog, code and test that ticket – in preparation for integration and deployment to live.

As per Figure 2 below, the Cycle Time metric view allows teams to understand time spent in each ticket status within the development cycle.  Analytics tools that offer filtering enable analysis by Status, Issue Type, or Epic (and any other standard or custom ticket field) all plotted over any time range required.

Figure 2 – Example Cycle Time metric view

Deployment Frequency is another fundamental measure of an organisation’s agility (when viewed alongside the other critical metrics described here).  A core objective of Agile delivery is the ability to develop and deploy live small software increments rapidly.  Deployment Frequency tracks that base competence and is a powerful metric around which to focus effort at all levels in the delivery organisation at the early stages of an Agile transformation. 

Figure 3 – Example Deployment Frequency metric view

Delivered Story Points is often considered a problematic metric due to the potential inconsistencies in the calculation of story points and how much effort they represent.  However, as a basic measure of output and how that is changing over time, it is a powerful metric around which to align. 

There may be concerns of teams ‘gaming’ the metric with story point inflation, but as with all metrics, they should be viewed in context by experienced folks who know the teams well.  And if this is the case, they can stil give an excellent view of how the delivery organisation is progressing over time.  

Figure 4 – Example Delivered Story Points metric view

And finally Escaped Defects is a simple but effective measure of overall software delivery quality.  It can be tracked in a number of ways, but most involve tracking defects by criticality/priority as per the example below. 

Figure 5 – Example Escaped Defects metric view

When these four simple Agile delivery metrics are viewed together, the early stage Agile DevOps practitioner can get a good balanced view of how their Agile DevOps maturity is progressing.

The metrics can be tracked over time, making sure that an improvement in one metric (e.g. Cycle Time) does not lead to a detrimental effect on another metric (e.g. Escaped Defects).  

In addition, the relationship between Cycle Time and Deployment Frequency can be closely watched.  Very often teams are able to reduce their Cycle Time, but this does not translate into quicker value delivery, due to bottlenecks in the integration and deployment process.  

Metrics for intermediate Agile DevOps practitioners

As organisations progress in their Agile transformation, they may start to consider a more complete set of Agile delivery and DevOps metrics.

Our suggestion would be that you may start to implement a hierarchy of cascading metrics.  These can be separated into:

1. ‘North Star’ leadership metrics  - to be adopted by the leadership team to set the overall direction for the delivery organisation 
2. Team and competence metrics – to be adopted by key players within the delivery organisation such as Team Leads, Delivery Managers, Product Managers, Engineering Managers and DevOps Managers.  These are relevant to the areas in question and help drive improvement in the aggregate ‘North Star’ metrics adopted by the technology leadership team.  

We would suggest that the early-stage metrics (discussed above) provide a sensible start point of ‘North Star’ metrics for the ‘intermediate’ Agile DevOps practitioner as they underpin the fundamentals of increased agility and are relatively easy to understand and act upon.

Figure 6 – Intermediate stage Agile DevOps Maturity metrics

The Team and Competence metrics have been selected as powerful determinant metrics that directly drive the ‘North Star’ leadership metrics that track your core progress towards Agile DevOps maturity. 

We have selected three metrics that in our experience most directly increase velocity (and hence reduce Lead and Cycle Time).  These are:

1. Flow Efficiency (which looks at the proportion of time tickets spend in an ‘active’ versus ‘inactive’ status)
2. Mean Time to Resolve Pull Requests (hrs)
3. First Time Pass Rate (%).

Typically these metrics are adopted by each scrum team (or squad) and related Scrum Masters and Delivery Managers, so that they are tracked and analysed in daily stand-ups, sprint retrospectives and management review meetings.

The Flow Efficiency analysis (see Figure 7 below), enables Team Leads to isolate and analyse each ‘inactive’ status in the workflow and consider if there is scope to reduce or eliminate it. The analysis shows the relative size of each ‘inactive’ status opportunity in terms of time spent in the inactive state and volume of tickets affected. 

Figure 7 - Example Flow Efficiency metric 

Typical opportunities to remove inactive bottlenecks include time spent with tickets awaiting definition (e.g. Sizing) and tickets awaiting QA.  Where waits for QA are considered excessive,  Delivery Managers can reconsider QA resource allocation by team.

Mean Time to Resolve Pull Requests (MTRPR) is often also found to be a key bottleneck and hence potential area to save time and reduce overall Cycle Time.  Very significant variations in time to resolve PRs are often seen between teams and individuals, with waits of over 50 hours not uncommon. 

Drill-downs can be used to understand variances by code repository and destination branch (see Figure 8 below).  This enables quick identification of the biggest bottlenecks and targeted intervention, with the result that MTRPR can be reduced dramatically (by approximately 50%). This has a very significant impact on overall Cycle Time.

Figure 8 – Example Mean Time to Resolve Pull Request metric 

In keeping with the ‘North Star’ metric of increasing Deployment Frequency, DevOps practitioners can track a range of metrics including: Number of Builds, Build Failure Rate, and Deployment Cycle Time.  All three are simple metrics which directly impact Deployment Frequency.

Figure 9 - Example Build Failure Rate metric

Our experience shows that typically you can expect to increase deployments per day (per pipeline) by 15% through a better understanding of the root-cause of Build Failures and of Deployment Cycle Time.

Delivery Team Leads and Managers can adopt a range of determinant metrics that help track and drive throughput and the delivery of value.

These include Stories Delivered by Epic, Lead Time for Stories and Epic, and Delivered Value Points (see Figure 10 below).

Figure 10 - Example value delivered metrics

And finally quality should also be a consistent focus - both the security and quality of the delivery process itself and the quality of the software delivered. 

We have selected a few simple Team and Competence metrics that directly impact defect rate - to reduce time spent fixing P1 (high priority) bugs, to improve the customer experience and reduce time diverted from feature development.

Real time analytics enable each team to focus on their own P1 resolution time and to better manage the backlog of Unresolved P1 and P2 bugs and time to resolve key hot fixes.

Figure 11 - Example quality metrics: P1 Resolution Time and Unresolved Bugs

We recommend that organisations at an intermediate level of Agile maturity start to adopt a broad view of ‘quality’ to include both the software output and the quality and security of the delivery process itself.   As such teams may also track and manage Commits without a Pull Request and Commits without a Ticket Reference.  See Figure 12.

The former ensures that all code is peer-reviewed before being committed (an important security requirement) – and the latter ensures the clear linkage between committed code and  tracking tickets, for security compliance. 

Figure 12 - Example delivery process quality metrics

Metrics for advanced Agile DevOps practitioners

As organisations reach an advanced level of Agile capability, they will no doubt have their own favoured metrics and analytics.  With that in mind, we have selected our favourite metrics for mature agile delivery organisations.  

Again, our suggestion is that you maintain a hierarchy of cascading metrics, separated into North Star’ leadership metrics to set the overall direction for the delivery organisation; and Team and Competence metrics – to be adopted by key players within the delivery organisation such as Team Leads, Delivery Managers, Product Managers, Engineering Managers, and DevOps Managers.  

As an example, mature agile delivery organisations are often very focused on refining the CI/CD process to increase deployment frequency and hence regular delivery of value to the organisation.   As such, additional metrics such as Failed Build Recovery Time and Mean Time for Failed Builds become popular as teams try to track and reduce the impact of build failures.  Flakiest Files is a DevOps metric which enables DevOps Managers to identify fragile source code files in their codebase which can then be targeted for refactoring to reduce failed builds. 

Figure 13 – Advanced stage Agile DevOps Maturity metrics

Figure 14 – Example Mean Failed Build Time metric- showing analysis by workflow name

About the Author

Charlie Ponsonby stared his career as an economist, helping developing world Governments shape their trade policy, working widely in the Far East, Africa, South Asia, and the Middle East. He then worked as a strategy consultant for Andersen Consulting (Accenture), before spells in senior leadership positions in retail, interactive TV and media.  He founded Simplifydigital in 2007, which was three times in the UK Sunday Times Tech Track 100 and grew to become the UK’s largest indirect sales platform for broadband and cable TV.  Charlie co-founded Plandek in October 2017, an end-to-end delivery metrics analytics platform, to help technology teams better manage delivery risk and improve their delivery effectiveness. Plandek is used by clients globally.  Charlie is married with three teenage children and lives in London.