Proper Usage of Metrics with Flow Debt as an Example

One of the big problems with software development at present is that it assumes it is manufacturing a product when actually it needs to think of it as creating a service based ecology.
The key difference between a software development process and a factory production line is that every work item is solving a different problem and hence is different from the others. Software development is building unique things. That uniqueness is what customers in general consider valuable.
Yet we see manufacturing models being applied to a non-manufacturing process such as software development. The notion that we need to drive variability out of the process is expressed in Lean Manufacturing and unfortunately it is leaking into the Agile software development. We see people constructing Statistical Process Control charts for Lead Time to see if the development process is under control. It is the belief that if we just eliminate variability the work will flow smoothly and all our problems will be over.
Variability is the natural companion of new product development. Eliminating variability eliminates innovation. We must understand the specific conditions that produce variability and manage our process to create these conditions. We need development processes that thrive in the presence of variability.

Metrics should be used in context. Measuring a complex software development process on the assumption of order is a fundamental error.

Let’s pick the Flow Debt metric as an example how metrics can be used improperly i.e. out of their domain. To the same effect we could have used other metrics such as Lead Time scatterplot, but Flow Debt is an important metric that deserves widespread usage.

What is Flow Debt?

Flow Debt is a leading indicator that provides a view of what is happening inside a delivery system. Flow debt is incurred when Lead Time defined as (Completed Date – Started Date) is artificially reduced for some work items in progress by “borrowing” Lead Time from other work items in progress. The term was coined by Daniel Vacanti in his excellent book Actionable Agile Metrics for Predictability: An Introduction.

Here is a flow debt example. Let’s say we have only one work item in progress. If we start another work item before finishing the first one, then we will have two in progress. If we finish the second work item before the first one, then we have incurred flow debt.

Usually people incur Flow Debt because of:

• Blockers i.e. impediments that block the flow of work. If we don’t want to be idle we will start on a new work when our current work is blocked.
  What is the difference between a blocker and a waiting state? Let’s use the analogy with my daily commute. The road is the workflow. The work item is my car. Each traffic light is part of the workflow and a potential waiting state. Such waiting is expected and I can’t claim I was late because a traffic light turned red! But if there is a car crash accident my car will be blocked. That is a blocker because it is unexpected, it has its specific cause and resolving a blocker requires some work to be done. In that case the traffic police should come and help restore the flow of cars.
• Policies that allow for too much WIP i.e. multitasking. Multitasking means working on multiple work items at the same time. In reality we constantly block and unblock work items in order to focus on another piece of work in progress.
• Inconsistent applications of pull policies. A pull policy determines the order in which work items are pulled into a delivery process once those items are committed to. In Kanban such a policy is called Class of Service. In reality most of the teams don't follow any pull policy at all. They just react to events and block work in order to make way for other "more important" work. Unless they know why to use a particular pull discipline they should stick to FIFO or FIFS.
• Parallel work. Some work items are just bigger (size as defined in Probabilistic Project Sizing Using Randomized Branch Sampling) than others. While we work on a big item we can finish several small work items in parallel and thus incur flow debt.

How to calculate Flow Debt?

One way to calculate Flow Debt for a given reporting interval is:
Flow Debt = The Approximate Average Lead Time (as predicted by the CFD) minus the exact average Lead Time for the items that finished.
By this calculation, a positive number means we are accumulating flow debt while a negative number means we are paying it off. However when we use it the sign will be reversed to show Flow Debt as negative.

Repaying Flow Debt

Flow Debt is repaid with less predictability for your process in the form of “longer than normal” Lead Times for the “artificially aged” items already in progress. The result is you can’t have confidence in your average Lead Time you thought you were capable of.
Flow debt is an indicator of predictability.
Predictability is the ability to make a quantitative forecast about a future state. Here we’ll limit ourselves to the ability to make forecasts about the completion times for a given work item – feature, epic, project etc. A time forecast includes date range and probability of occurring – e.g. less than 3 days with 85% probability. A forecast without an associated probability is nonsense. The narrower the date range the more predictable we are. For instance, having lead time of less than 10 days with 85% probability is more predictable than having lead time less than 20 days with 85% probability.

Predictability is not the same as speed. One can be predictably fast but also predictably slow. But only when we are predictable we can start looking for improving our speed of delivery.

Metrics in context

Operating in a highly constrained, highly controlled environment is extremely useful when we are looking for predictability. However, we should not be assuming order where order is practically impossible. Measuring a complex software development process on the assumption of order is a fundamental error. Metrics should be looked at in context as well. Complex stuff should not be measured together with the Obvious. To do that first we need to know what type of problems we are dealing with - ordered, complex or chaotic. The Cynefin framework can help us do that.

The Cynefin framework is a sense-making framework.  Sense-making is the way humans chose between multiple possible explanations of sensory and other input as they seek to confirm their perceptions with the reality in order to act in such a way as to determine or respond to the world around them [2]. The framework is first about contextual awareness, followed by contextually appropriate action.

As described in detail in the InfoQ article Adaptable or Predictable? Strive for Both – Be Predictably Adaptable! we create two contextualized Cynefin frameworks – one for the Client and another for the Capability (represented by the delivering organization’s knowledge, skills and capacity) contexts.

The uncertainty about the nature of the client requirements defines the Cynefin domains for the Client’s context. We will use only three cases:

1. The client knows exactly what they need. They have it already defined. (Obvious)
2. The client knows what they need but not exactly. They’ll need some expert analysis in order to define it. (Complicated)
3. The client has just a vague idea about what they need. They’ll have to explore several alternatives before arriving at a definition. (Complex)

The uncertainty about if the development capability would match the client requirements defines the Cynefin domains for the Capability’s context. Again three cases:

1. The organization has all the knowledge and skills required to do the job. (Obvious)
2. The organization has some of the knowledge and skills required to do the job. The rest we’ll research and analyze because someone somewhere has done it before. (Complicated)
3. The organization has none of the knowledge and skills required to do the job. We haven’t done it before and we’ll need to explore alternatives and build knowledge. (Complex)

We match the client requirements from each Cynefin framework indexed by their domain. The resulting Capability / Client matrix represents the Uncertainty about the work items we'll need to deliver.

If a requirement is part of a Complex domain, it is considered Highly uncertain. If a requirement is part of a Complicated domain, it is Medium uncertainty. Low uncertainty are only requirements which are Obvious in both frameworks.

Proper usage of Flow Debt

Let’s see an example of a proper i.e. context aware usage of Flow Debt with an IT operations team. They offer Identity & Access Management services related to internal employee and contractor user life cycle.
Let’s categorize service requests using the Cynefin based approach we outlined above.
First to contextualize from client’s perspective – do clients know what they need when they file a service request? Indeed, in majority of the cases they do.  They need to be provisioned access to different applications. 
Next to contextualize from capability’s perspective – does the IT operations team know how to fulfill a service request? Indeed, they do. Obvious access provisioning requests are fulfilled according to the best practices from team’s Knowledge Base. Complicated requests are fulfilled by utilizing the expertise available inside and outside the team. Very few requests are Complex – those are transferred to another specialized team.
Based on the above contextualizations we can categorize the Service Requests as Low and Medium uncertainty.

As outlined in the previously mentioned InfoQ article on being predictably adaptable,  obvious problems have known solutions based on best practices. Complicated problems have predictable solutions, but require expertise to understand, so there are multiple good practices. Hence work items categorized as Low and Medium uncertainty can be delivered predictably.
Flow debt is an indicator of predictability. We can use it in the context of this IT operations team to check are they predictable.

Their delivery rate is visualized on the below Cumulative Flow Diagram (CFD).

(Click on the image to enlarge it)

Across the X-axis are the dates from the reporting period. The Y-axis shows the cumulative count of work items in different states of the delivery process. The flow of work is represented by the cumulative number of started work (the top of the orange band) and the cumulative number of deliveries (the top of the blue band). In order to ease comprehension, the CFD does not have the Backlog items shown, but only the work items started and delivered. The steeper the blue band the more work items delivered per day of work.

But from the CFD we cannot read if the flow of work is taking more time than “normally” expected. 

The below Flow Debt diagram does provide the insight the CFD doesn’t. Time is in weeks.

The red bars on the diagram show when Flow Debt is incurred. The green bars show when Flow Debt is repaid. We see that the team was in the Flow Debt most of the time. The effect is visible on the Lead Time Scatterplot bellow.

(Click on the image to enlarge it)

Across the X-axis is a timeline of the underlying process as represented by our data. The Y-axis represents the Lead time again from our data in days.
Plotting a dot for every work item completed for the reporting period generates the scatterplot. We do that by finding the date at which the work item was completed and plot a dot on the chart according to its Lead Time. If there are several work items that finish on the same day with the same Lead Time then we simply plot several dots on top of one another.

Based on that we can reason their delivery time is unpredictable. Indeed, that was the feedback from their clients who were are experiencing delays in access provisioning resulting in lost productivity and cost to the business. There was a great deal of dissatisfaction and frustration with the services that the team provided.

The huge amount of flow debt the team incurred is actually good news! If they look into stop incurring Flow Debt i.e. establish pull policies, stop multitasking, decrease blockers resolution time they could become more predictable and deliver faster with no need to work harder.

Improper usage of Flow Debt

Now, let’s examine a case where using Flow Debt as a basis for management decisions could be misleading, even damaging. Back in the year 2011 this was a project to deliver Virtual Recruiter – a ChatBot, similar to present day’s Sgt. Star of the U.S. Army, able to answer on Facebook Messenger questions like “What Java jobs you have in Boston, MA?”. Functionality also allowed for applying for a job, scheduling job interviews, multi-language support etc.

Following the same approach as for the IT operations team let’s categorize the project requirements.
First to contextualize from client’s perspective – did the client know exactly what they needed when they asked for a Virtual Recruiter on Facebook? No, they didn’t. They had a vague idea about what they need – hence that’s a Complex domain problem.
Next to contextualize from development capability’s perspective – did the delivery team know how to develop a Virtual Recruiter on Facebook? No, they didn’t. At the time there were chatbots developed for different purposes, but not a Virtual Recruiter on Facebook. Also there was no established way to develop a chatbot, but many competitive technologies and architectures. The team had to select a technology and design the architecture. That kind of work definitely belongs to the Complex domain.
Based on the above contextualizations we can place majority of the project work into the High uncertainty category. It is important to note that not all work items, part of the project scope, were High uncertainty. Many of them were Medium uncertainty but the core of the project was indeed High uncertainty.

The fundamental dynamic for software development is shifting a High uncertainty problem from complexity to order and maintaining it there in such a way that it becomes predictable.
That’s why we schedule the High Uncertainty work items first. They need exploration and there is a high probability they may negatively affect the schedule. At the same time there is a high probability we get lucky and finish them much earlier than expected.  We employ numerous coherent, safe-to-fail, fine-grained experiments in parallel which will provide feedback on what works and what doesn’t. We manage the experiments as a portfolio.
The Low and Medium uncertainty items we schedule after the High items. Not doing that would mean we try to hide uncertainty under the carpet.

The above dynamic is illustrated on the below CFD. Again the flow of work is represented by the cumulative number of arrivals (the top of the red band, very tiny as we can see) and the cumulative number of deliveries (the top of the blue band). 

The delivery rate follows a sigmoid or Z-curve pattern. We can see that initially the delivery rate was low. It was because the team started by tackling the High uncertainty requirements first e.g. running experiments in order to select the technology to be used and the system architecture to be implemented. Those experiments and the decision making associated with them took significant amount of time. When in mid-September the team managed to move the core problems into the Complicated domain the delivery rate increased dramatically.

Let’s imagine the project sponsors have read Daniel Vacanti’s excellent book Actionable Agile Metrics for Predictability: An Introduction. Inspired by the book they decide to use Flow Debt as a metric to see if the team is predictable and ultimately if the project is managed in a way to ensure predictable delivery.
If they have created the diagram on September 1^st they would have seen the image below.

They would have been alarmed because Flow Debt diagram shows that for the first three months the actual average Lead Time was almost four weeks more than the “normal” average Lead Time! The sponsors may even consider canceling the project because it looks out of control!

Of course they would be wrong because they used a metric such as Flow Debt unaware of the context. We now see how using Flow Debt out of context to judge project progress would be misleading and even damaging!

Since we are aware of the context we know that such level of Flow Debt is expected because the High uncertainty work items were scheduled first and run as a portfolio of parallel probes. Running things in parallel by definition will incur Flow Debt. As we see our management approach, informed by the Cynefin framework, mandates high level of Flow Debt during initial part of the project!

The CFD shows that the delivery rate changed in September.  Indeed, as we see on the below diagram on September 6^th the Flow Debt was repaid, the team finished with the probes and switched to development i.e. focused their attention to the Medium and Low uncertainty work items.

After that as we see that the team managed the level of Flow Debt as it is expected when working on Medium and Low uncertainty work.

This project does tell a story and that is one of massive upfront uncertainty turned into predictable performance. A perfect example of being predictably adaptable!

Conclusion

The thrust of the article is to present applications to software development of the ideas from complexity theory and the Cynefin framework in particular.
Looking through the Cynefin lens we saw how to properly use metrics using as an example a flow metric called Flow Debt.

References

1. Vacanti, Daniel S. Actionable Agile Metrics for Predictability: An Introduction, Daniel S. Vacanti, Inc. (2015)
2. D.J. Snowden, Multi-ontology sense making: a new simplicity in decision making Informatics in Primary Care, 13 (1) (2005), pp. 45–54

About the Author

Dimitar Bakardzhiev is an expert in managing successful and cost-effective technology development. As a Lean-Kanban University (LKU)-Accredited Kanban Trainer (AKT) and avid, expert Kanban practitioner and  Brickell Key Award 2015 Finalist, Dimitar puts lean principles to work every day when managing complex software projects.
LinkedIn https://bg.linkedin.com/in/dimitarbakardzhiev
Twitter @dimiterbak