The Importance of Metrics to Agile Teams

The importance of metrics to Agile teams

We are fortunate to have the opportunity to work with a great variety of engineering teams – from those in start-ups to very large, distributed enterprises. 

Although definitions of “engineering excellence” vary in these different contexts, all teams aspire to it. They also share the broad challenge of needing to balance the “day job” of delivering high quality, high value outcomes against the drive to continually improve. 

Continuous Improvement (CI) inherently requires metrics against which to measure progress. These need to be balanced and meaningful (i.e. deterministic of improved outcomes). This creates two immediate issues:

• First - metrics (and indeed the concept of measurement) are contentious. What is the ideal “balanced scorecard”? Is there even such a thing?
• Second – the Agile philosophy is predicated on decentralised, empowered and self-managing teams, which runs counter to the concept of top-down measurement. 

We view CI as vital in healthy and maturing Agile environments. Hence metrics to underpin this process are also vital. However, CI should be owned and driven by the teams themselves so that teams become self-improving. Ergo, CI programmes become SI (Self-Improvement) programmes.

This article focuses on how teams can implement a demonstrably effective SI programme even in the fastest moving and most resource constrained Agile environments so that they remain self-managing, deliver value quickly, and continue to improve at the same time.

The Size of the Prize

The concept of CI has been around for a long time. It was applied perhaps most famously in a business context in Japan and became popularised with Masaaki Imai’s 1986 book “Kaizen: the Key to Japan’s Competitive Success.” 

The CI principle is very complementary with core Agile principles. Indeed, the Agile Manifesto states:

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly.

There are two key themes here – firstly, CI and secondly, that CI is driven by the teams themselves (SI). This raises the question as to what role of leadership should take in this improvement process. 

Our evidence shows that the size of the prize is very significant. Well implemented SI programmes can deliver significant and sustained improvement in metrics that underpin your time to value (TTV) – for example:

• 10%+ velocity improvement
• 10%+ improvements in flow efficiency
• 15%+ reduction in return rates and time spent reworking tickets (returned from QA)
• 30%+ improvement in sprint completion accuracy (Scrum Agile) 
• Greatly improved team collaboration and team wellness.

However, achieving these goals is hard and requires sustained effort. Technology leadership needs to give teams the tools (and encouragement) to own and drive the self-improvement process. Without constant support, many teams will not have the time or inclination to drive their own self-improvement as they strive to meet their short-term delivery objectives.

The tools needed for effective Agile team Self-Improvement 

The principle of team Self-Improvement (SI) is simple and powerful, but very hard to deliver effectively. It requires four important things:

1. A serious long-term commitment and sponsorship from both the leadership team and the teams/squads themselves – and requires effort and resources over a prolonged period of time to realise iterative improvement
2. An agreed, objective set of metrics to track progress – making sure that these metrics are actually the right ones, i.e. deterministic of the desired outcome
3. A means for teams to easily track these metrics and set targets (with targets calibrated against internal and external benchmarks)
4. An embedded process within teams to make the necessary changes; celebrate success and move on.

Agile teams are almost always busy and resource-constrained. As a result, the intention of always improving (in a structured and demonstrable way) often loses out to the pressures of the day job – delivering to the evolving demands of the business. 

In our experience, successful SI requires coordination and stewardship by the technology leadership team, whilst empowering teams to own and drive the activities that result in incremental improvement. Therefore this needs to be in the form of a structured, long-term and well implemented SI programme. 

Implementing an effective team Self-Improvement programme

Self-Improvement needs a serious commitment from the leadership team within engineering to provide teams with the tools they need to self-improve. 

This will not be possible if the organisation lacks the BI tools to provide the necessary metrics and reporting over the full delivery lifecycle. Firstly, the reporting found within common workflow management tools like Jira is not optimised to provide the level of reporting that many teams require for an effective SI programme. Secondly, teams use a number of tools across the delivery cycle, which often results in data existing in siloes and not integrated to reflect a full view of end-to-end delivery.
Teams should seek out BI tools that address these challenges. The right tools will give product and engineering teams meaningful metrics and reporting around which to build robust SI programmes.

Metrics for SI

As mentioned in the intro, selecting and agreeing metrics is often the most contentious issue. Many programmes fail simply because teams could not agree or gain buy-in on meaningful sets of metrics or objectives.

By its very nature, Agile encourages a myriad of different methodologies and workflows which vary by team and company. However, this does not mean that it’s impossible to agree achieve consensus on metrics for SI. 

We believe the trick is to keep metrics simple and deterministic. Complex metrics will not be commonly understood and can be hard to measure consistently, which can lead to distrust. And deterministic metrics are key as improving them will actually deliver a better outcome. 

As an example – you may measure Lead Times as an overall proxy of Time to Value, but Lead Time is a measure of the outcome. It’s also important to measure the things that drive/determine Lead Times, levers that teams can actively manage in order to drive improvements in the overarching metric (e.g. determinant metrics like Flow Efficiency).  

The deterministic metrics we advocate are designed to underpin team SI, in order to steadily improve Agile engineering effectiveness.

The (determinant) metrics are grouped into six key areas. These are:

1. The key enabler – best practice and tool use 

   A key push-back is often that tool usage (e.g. Jira) is so inconsistent, that the data collected from within it is not meaningful (the old adage of “garbage in, garbage out”).

   However, there are some simple disciplines, that can themselves be measured, that greatly improve data quality. 

   In addition to focusing on best practice “hygiene” metrics, teams can build their self-improvement initiatives around five further determinant metric sets…

2. Sprint disciplines and consistent delivery of sprint goals (Scrum Agile)
3. Proportion of time spent/velocity/efficiency of writing new features (productive coding) 
4. Quality and failure rates and therefore…
5. Proportion of time spent/efficiency of bug fixing and re-work
6. Teamwork, team wellness and the ability to collaborate effectively. 

From these six areas, we believe these are some of the most common and meaningful metrics around which a team can build an effective self-improvement programme:

In our experience, a highly effective Agile SI programme can be built around these metric sets. We’ve also found that having an integrated view of the full delivery cycle across the right tools in a single view, underpinned by these core metrics reveals key areas that can be optimised, i.e. low hanging fruit that can materially improve Time to Value. 

Metrics should be available in near real-time to the teams, with minimal effort. If teams have to collect data manually, the overall initiative is likely to fail. 

A sample SI Dashboard

[Click on the image to enlarge it]

When all team members have a near real-time view of the metrics that they’ve signed up to, these become a core part of daily stand-ups and sprint retrospective reviews. 

The aim is not to compare these metrics across teams - instead the key aim is to track improvement over time within the team itself. Leadership teams need to remain outcome focused, whilst enabling and empowering teams to identify and make incremental improvements that will improve those outcomes.

Running the SI programme

Team SI is unlikely to take place consistently and sustainably across teams, without committed leadership. The SI programme needs to be formally established on a monthly cycle of team target-setting, implementation, review, and celebration of success (see below).

Team Leaders and Scrum Masters need to strike the right balance of sponsoring, framing and guiding the programme with giving teams the time and space they need to realise improvements. 

SI is designed to be a positive and motivating process – and it is vital that it is perceived as such. A key element of this is remember to celebrate success. It’s easy to “gamify” SI and find opportunities to recognise and reward the most-improved teams, competence leaders, centres of excellence, and so on. 

Target setting

Questions often arise around target setting and agreeing what success looks like. Some organisations opt only to track individual teams’ improvement over time (and deliberately not make comparisons between teams). Still others find benchmarks useful and divide them into three categories:

1. Internal benchmarks (e.g. measures taken from the most mature Agile teams and centres of excellence within the organisation) 
2. External competitor/comparator benchmarks – some tools provide anonymised benchmarks across all metrics from similar organisations
3. Agile best-practice benchmarks - these are often hard to achieve but are obvious targets as the SI programme develops.

The SI programme leader/sponsor can view progress against these benchmarks and look back over the duration of the programme to view the rate of improvement. 

In summary:

• The philosophy of Continuous Improvement is central to Agile. The Agile Manifesto states:

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly. 

• Continuous Improvement (CI) should not be imposed and driven top-down – instead it should be led by Agile teams themselves, so Self-Improvement (SI) is a more suitable terminology
• he evidence shows that Agile engineering teams practicing SI perform significantly better than Agile teams simply focused on their immediate delivery priorities. 
• However, effective and sustained SI is hard. It requires: 

1. formal sponsorship by technology leadership in the form of recognition and a suitable framework to manage the long-term process; and crucially
2. a set of meaningful and agreed Agile metrics that underpin the process of SI and track performance improvement over time; and
3. a means to surface these metrics in near real time, with minimum/no effort involved for the teams themselves.

About the Authors

Will Lytle, Director, Customer Success, Plandek. Will is a senior technologist, specialising in digital transformation and leading cross-functional delivery teams. He has more than 15 years of global experience in Agile and Hybrid development, operating models, developing talent, systems integration and production management. In his current role as Director, Customer Success for Plandek, Will helps agile teams to deliver more effectively by applying metrics for continuous self-improvement.Will is an avid runner and cyclist, and is currently training for a marathon. He volunteers for ‘Inspiring the Future,’ a UK charity that helps children from disadvantaged backgrounds realise their full potential.

Charlie Ponsonby is Co-CEO at Plandek. Charlie started his career as an economist working on trade policy in the developing world, before moving to Andersen Consulting’s Strategy practice in 1992. In 1996 he joined the Operating Board of Selfridges, before moving to Open interactive TV in 1999 and then Sky in 2001 where he was Marketing Director until leaving to found Simplifydigital in 2007. Simplifydigital was three times in the Sunday Times Teck Track 100 and grew to become the UK’s largest TV, broadband and home phone comparison service, powering clients including Dixons-Carphone, uSwitch and Comparethemarket. It was acquired by Dixons Carphone plc in April 2016. He co-founded Plandek with Dan Lee in October 2017.