Common estimation approaches often fail to give us the predictability we want. Forecasting provides a range of possible outcomes with the chance of outcomes becoming reality. It can answer questions like “When will it be done?” or “What can we deliver by xx?” with confidence.
Mattia Battiston spoke about agile metrics for predicting the future at Aginext.io 2020.
Estimates on story size rarely predict how long things will take; factors like having a high amount of work in progress, the time spent in queues between activities, and unexpected events have a much bigger impact on lead time, Battiston explained.
Forecasting uses the team’s historical data to simulate what might happen in the future. Predictions are expressed as a range of possible outcomes. Expressing outcomes with a chance of becoming reality changes our thinking from deterministic to probabilistic, Battiston mentioned. It enables us to have conversations about why things take long, what impacts lead time, and what risks are influencing delivery.
InfoQ interviewed Mattia Battiston, a software developer and team leader at Tesco, about using historical data and metrics to forecast the future.
InfoQ: What is it that causes common estimation approaches to often fail to give us the predictability we want?
Mattia Battiston: Let’s take the most common estimation technique as an example: story points (but the same applies with t-shirt sizes, ideal hours, and any other technique where we ask people to give their best guess). Many teams observe that there is little correlation between the estimated size of a story (number of story points) and how long the story actually takes to be completed (lead time). In a team where I worked, for example, stories estimated with the same number of points were taking from a few days to a few weeks to be completed.
Low correlation between story points and lead time in Mattia’s team
There are three important factors that have a much higher impact on lead time than the story size, and that when left unmanaged make our teams unpredictable.
First, do we have a high amount of work in progress (WIP)? When we work on too many things at the same time we are not able to focus on finishing the tasks that are already in progress. We waste time in context switching, the quality of our work decreases, and even stories that appear to be simple end up taking longer than expected.
Second, how long does work spend in queues between activities? Very often in our processes there is some waiting time between one activity and another (for example, waiting for a developer to be free to start a story, waiting for the next release, etc). These queues are often invisible, they’re not represented on our boards, and it’s really common to ignore them when we estimate, as we only tend to consider the active time that we’re going to be working on something. When these queues are not managed they lead to a lot of work in progress put on hold, which in turn leads to high unpredictability.
Example of queues in our process
Third, are unexpected events common? Events like being blocked by a broken test environment, having to clarify requirements, or urgent problems such as a defect in production make us interrupt and put on hold activities that have already been started.
When we have one or more of these problems - and most teams I’ve worked with certainly seem to - then our estimates don’t give us much predictability because they have a low correlation with the actual time taken to complete the work.
If you’re curious to find out how predictable your estimates are, you can use the spreadsheet are my story points predictable?
InfoQ: What’s your approach for estimating to predict the future?
Battiston: When trying to predict the likelihood of events with high uncertainty, such as how long it will take to complete a piece of work, I coach the teams I work with to use forecasting techniques.
A forecast typically uses the team’s historical data to simulate what might happen in the future. Predictions are expressed as a range of possible outcomes, together with the chance of that particular outcome becoming reality.
Example: the forecast for a project (starting in Iteration "I334")
Troy Magennis, who is one of the gurus in the Agile metrics field, once taught me what I consider to be the real hidden power of forecasting: "Forecasting is a communication field". It’s all about improving how we communicate our predictions.
By expressing the results like this, we change our thinking from deterministic, aka "We can predict the future", to probabilistic, aka "These are all possible outcomes, some more likely than others". This encourages great conversations like, "What if the project did take this long? What would be the impact of that? What can we do to reduce this risk?"
InfoQ: How can we do forecasting?
Battiston: The mechanics vary depending on what business question we are trying to answer, but the principles remain to use historical data and express the results in a probabilistic way. Here are some examples.
"How long will this story take?" Collect the lead time of your past stories, and calculate a few percentiles, e.g. 50th, 80th and 90th.
Then, the answer would sound like this: "We are 80% confident that it won’t take longer than 9 days. It might take just 4, but there is only a 50% chance so I wouldn’t make any promises on that. In the worst case scenario we might need 10 days, but there’s only 10% chance that this might happen".
We are setting expectations that both the team and the stakeholders can be confident with. It’s tempting to simply pick a number with a high confidence and answer "9 days", but we much prefer being clear with our stakeholders and explain that the only realistic answer is a range of possibilities.
"How much work can we do in the next Sprint?" Similar to the above, we can collect the past throughput of the team (aka how many stories they’ve completed for each Sprint) and calculate some stats.
Then, the answer would sound like this: "We’re 80% confident that we can complete at least 5 stories, because in the past we have completed at least that many stories in 80% of the Sprints. We might get up to 8 stories, but there’s only a 50% chance, so we probably shouldn’t make any strong commitment on that."
For more examples and details you can look at the book that Chris Young and I are co-authoring, the Team Guide to Metrics for Business Decisions. We also published a few spreadsheets that you can use to put these ideas in practice: Public resources for the book Team Guide to Metrics for Business Decisions.
InfoQ: What data do you collect and how much do you need?
Battiston: You can get started with simply recording the start and end date for each story. That’s enough to calculate its lead time, and to count how many stories are completed per period of time (the so/called "Throughput" or "Delivery Rate").
You don’t need much data to start: 5 samples is a good "minimum", and 7-11 samples are enough to give you good accuracy.