Monte Carlo Planning Improves Decision Making

In 2010 I helped a startup IPO by creating a custom Monte Carlo planning tool in C# that estimated the likelihood that their new small business product would ship on schedule. The output of this tool -- a probability distribution over possible release dates -- helped the software development teams and management make tradeoff decisions. When the product was released on time, just a few weeks before the IPO, the CTO and founder of the company said that it was the first product in the history of the company that was on time. Prior to Monte Carlo planning, the company had been using standard project management and agile planning approaches.

In addition to improving estimation, Monte Carlo planning also helped to change the culture and thinking around planning. When C-level executives asked for a date the VP of Engineering was now able to have intelligent conversations about the confidence level of that date. The VP of Engineering began to say, “A release is not a date, it’s a probability.” This in turn led to significant changes in what metrics were tracked (from velocity to end-to-end inventory management) and how the product management group thought about risk.

This is one of many examples of Monte Carlo leading to better results and decisions. Monte Carlo planning generate the full probability distribution based on historical data. In contrast, the standard agile planning approaches generate a straight line average case analysis.

A simple example of Monte Carlo planning

Let’s walk through a simple example of Monte Carlo planning to illustrate its benefits.

Consider a Scrum team that has a velocity that ranges from 80 to 120 (uniformly distributed). The Scrum team wants to know how many points will be completed in a six sprint release. If the team simply computes its average velocity it will conclude that it can complete 600 points (6*100) over six sprints. In contrast, here are the results of a Monte Carlo simulation with 20 sample runs:

These results show that over six sprints there is a 10% chance that the velocity will average 92, a full 8% less than what the average case analysis shows. This data immediately leads to useful conversations with the Product Owner: “What will happen if the bottom 8% of the backlog is not completed by the release date? If that is acceptable, can the product be released earlier? If this is not acceptable, what options can we explore?”

How was this probability calculated? Monte Carlo samples the historical distribution, in this case a uniform distribution between 80 and 120. To create the graph above, I sampled the distribution 20 times. Here is one of the samples:

This sample consists of six sprints because the team wants to estimate over a six sprint release. In this particular sample, the velocity ranged from a low of 91 to a high of 120. The average was 108.5. In the chart above this sample is responsible for the bar at ‘108’ on the X axis.

Here is another sample:

In this sample, the velocity ranged from a low of 87 to a high of 113 with an average of 95.

Think of each sample as a ‘possible future.’ Monte Carlo creates these possible futures from historical information and correctly weights them.

One of the great benefits of Monte Carlo is that it works with any distribution, even ones that are unknown. In this simple example the distribution is known and the problem is simple enough that the probability distribution at the end of six sprints has a closed form solution. However, this is typically not the case.

A more complicated example of Monte Carlo planning

For problems that are encountered in real world situations, either the closed form solution is not possible or is so hard to derive that a Monte Carlo approach is faster. Here is an example based on an actual situation that I encountered at a client and is quite common:

Here are the key aspects of this model:

• There are two product development teams running in parallel. Each team’s velocity is 100 +/- 20 (uniform distribution).
• Each team must finish 575 points.
• Business Acceptance finishes within one week with 95% probability and within two weeks with 100% probability.
• The goal is to complete the effort in thirteen weeks: 6 sprints (12 weeks) in Development and one week in Business Acceptance

What is the probability that the project will be on time? Monte Carlo produces a result of approximately 45%.

Monte Carlo planning cannot be approximated by any straight line estimation

Given that average case analysis (such as the standard velocity based planning used by many Scrum teams) is easier than Monte Carlo, a natural question is whether or not there is any average case straight line assumption that will approximate the Monte Carlo results. To test this possibility we will see what happens under three velocity assumptions for this more complicated problem:

• Best Case assumption: Projected velocity = 120 points
• Average Case assumption: Projected velocity = 100 points
• Worst Case assumption: Projected velocity = 95 points

In all three cases Business Acceptance takes one week. Clearly, at the start of the effort all three estimates are far off. The best case and average assumptions indicate that there is a 100% chance the date will be hit while the worst case assumption indicates that there is a 0% chance that the date will be hit.

Given that the initial estimates are far off, how many sprints does it take for these assumptions to get close to the 45% probability that Monte Carlo produces from the start? Here are the results for 20 sample runs:

The chart shows that:

• Best Case analysis (blue line) over-estimates the probability of success until iteration 6.
• Average Case analysis (red line)  also over-estimates probability of success until iteration 3.
• Worst Case analysis (green line) under-estimates probability of success until iteration 4.

In short, there is no straight line approximation that comes close to the Monte Carlo results. Why is this? Confidence intervals change as the square root of time so any straight line approximation will be far off over any reasonably long time period. In addition, the average case analysis always produces a single result -- either the date will or will not be hit -- which is clearly incorrect.

Is there a way to tweak the straight line analysis to produce better results? Some agile experts suggest creating straight line approximations to produce confidence intervals. For example, taking the lowest three velocities out of the last five sprints produces a ‘pessimistic’ estimate while taking the highest three velocities out of the last five sprints produces an ‘optimistic’ estimate. The problem with these straight line approaches is that the probability that the actual results will be between the low and high straight line estimates changes as a function of time -- it is not constant. So these straight line ‘lower’ and ‘upper’ bounds are incorrect. As mentioned previously, confidence intervals change as the square root of time so no straight line approximation works except over very short time intervals.

Advantages of Monte Carlo

Here is why I like Monte Carlo simulations:

• They are a step towards an end-to-end simulation and understanding of a company’s operating and performance characteristics. Ultimately, Monte Carlo can help answer questions such as: Given the choice between doubling the number of teams or doubling the amount of time, what is better?
• Monte Carlo generates the full probability distribution for cost, time, and scope (or anything else) which improves decision making.
• Monte Carlo overcomes significant shortcoming in standard agile approaches to planning:
  • Velocity based planning practices typically do not work for multiple teams or for teams with dependencies because velocity says nothing about how long an item spends in a product backlog. And in the case of single teams velocity based planning is a straight line average case analysis which provides no measure of confidence or variance.
  • SAFe recommends voting to determine confidence in the PI (program increment) plan: “Once program risks have been addressed, teams vote on their confidence in meeting their program PI objectives.” While this human-centered planning approach is admirable, it is incomplete and deserves to be supplemented by a quantitative approach.
  • Dependencies are not addressed in a rigorous way.
  • Like all human estimates, estimated used in agile planning are subject to significant personal and organizational biases.

Concerns with Monte Carlo

Here are common concerns I hear about a Monte Carlo approach to planning:

• The future is different from the past, hence sampling from historical data produces poor results. This can be a very technical issue but I like to simplify it: Either the system is stable or it is not. If it is stable, then historical sampling works well. If the system is not stable then no forecasting approach will be accurate enough to satisfy management.
• Creating a Monte Carlo simulation is difficult. It is definitely harder than doing an average case straight line analysis. But it is getting easier with free Excel spreadsheets provided by Troy Magennis. In a section below, I describe how to use Excel to create a simple Monte Carlo simulation.
• Executives do not understand probability distributions. This is my experience as well. So instead of showing executives the entire probability distribution, I like to identify probabilities at key decision points and communicate like this: “There is a 15% chance that the initiative will miss your target date and you will not get your bonus.”
• The problem with estimates is not how they are produced but how they are interpreted. This issue is quite common no matter how the estimates are generated. However, when the estimation approach is not quantitatively rigorous this invites politicking and ‘management of expectations.’
• Monte Carlo does not work when there is no historical data. This is correct! In this situation, I often suggest that teams minimize the amount of time they spend planning and estimating and instead start working to generate historical performance data.

A final objection to Monte Carlo is that it does not address the lack of trust and alignment that organizations often have when contemplating plans and examining estimates. This lack of trust and alignment then invites management to set targets without regard to historical performance. This is certainly a valid worry and one that I suggest be handled separately. There are many alignment activities, such as handing out ‘value dollars’ and inviting stakeholders to buy features or deadlines, which will expose different assumptions and tradeoffs. Once everyone involved has the same understanding of the current reality and the goals then Monte Carlo planning might be appropriate.

Demonstrating the Inaccuracy of Human Estimates

One obstacle to embracing Monte Carlo planning is overconfidence in the accuracy of human estimates.

Here’s a simple technique that I use to demonstrate the accuracy or inaccuracy of human estimates:

1. Select a work item to be estimated. At the team level, this is often a user story. At the program level, this is an initiative or a project.
2. Select a group of experts that that will do the estimation. At the team level, this is almost always the people doing the work. At the program level, this will often be a group of senior engineers.
3. Split the group of experts into two and have them estimate the work item separately.
4. Define a measure of error. Here’s a simple one: Let m be the average of the two estimates and x2 be the larger of the two estimates. The metric is x2/m - 1. For example, if the two estimates are 80 and 120 then the metric is 20% (120/100 - 1).
5. Ask the participants how bad the metric can be before the cost of creating the estimate exceeds its value. For the metric defined in the previous step, the range is typically 10% to 25%.
6. Calculate the metric and debrief the results.

More often than not, the result shows that the error is greater than that needed to create a useful estimate. This provides a launching pad into a discussion of the value of Monte Carlo and addresses the concern that it is too difficult to do. At a recent training event at a 400 person company, I ran through these steps in approximately 30 minutes and the error rate was 33% (i.e., the high estimate was twice the low estimate) for a single user story.

How to Introduce Monte Carlo Planning

Monte Carlo planning may not be suitable in all situations. When it is, here is how I like to introduce it:

1. Understand existing planning practices.
2. Explain shortcomings of the existing approach and standard agile approaches.
3. Demonstrate benefits and differences of the Monte Carlo approach using a practical example from the company.
4. Quantify business benefits.
5. Expand the application of Monte Carlo planning.

Step 1 is often overlooked but is a critical part of introducing this change (and, really, any other change). Thankfully, I have never met a company which enjoyed its planning approach or thought the results were highly accurate. In fact, at a Fortune 500 company where I consulted one of the people in charge of annual planning told me that there was “no relationship between planning and actuals”! In these situations, the primary problem is getting historical planning data and modeling the system of work.

Get a taste of Monte Carlo planning with a simple spreadsheet

A simple Monte Carlo planning spreadsheet can be created in Excel using the randbetween and hlookup functions.

Here is what the formulas look like:

Cells A1, B1, and C1 contain the numbers 1, 2, and 3. Think of these as the last three sprints. Cells A2, B2, and C3 are the velocity the team achieve in each sprint.

Row 4 contains the formula

RANDBETWEEN generates a random number between 1 and 3 (because there are three sprints). Call this random number N. HLOOKUP then returns the velocity of the Nth sprint. So row 4 samples the historical distribution to create another three sprint sequence.

You can repeat row 4 as many times as you want to create histograms (probability distributions) and do other types of analyses. In the two examples I gave at the start of this article I repeated row 4 twenty times to create 20 samples. Most Monte Carlo studies use over 1,000 samples.

Summary

Monte Carlo planning samples the historical distribution to give a rigorous, quantitative account of what the future may bring.

It holds significant advantages over standard average case straight line agile approaches and you can start your journey to Monte Carlo with a simple Excel spreadsheet.

Additional resources

If you would like to further your study of Monte Carlo planning, I invite you to examine these additional resources:

• InfoQ article: #NoEstimates Project Planning using Monte Carlo Simulation by Dimitar Bakardzhiev.
• Video: Monte Carlo planning by Michael de la Maza.
• Presentation: Monte Carlo planning by Michael de la Maza (10 slides).
• Presentation: Monte Carlo planning by Michael de la Maza (20+ slides).
• Book: Forecasting and Simulating Software Development Projects by Troy Magennis.
• Website: Focused Objective (Troy Magennis).
• Website: Actionable Agile (Daniel Vacanti).

About the Authors

Michael de la Maza is a Scrum Alliance Certified Enterprise Coach (CEC). As an Agile consultant, his major engagements have been with Paypal, State Street, edX, Carbonite, Unum, and Symantec. Previously, he was VP of Corporate Strategy at Softricity (acquired by Microsoft in 2006) and co-founder of Inquira (acquired by Oracle in 2011). He is the co-author of Professional Scrum with TFS 2010 and Why Agile Works: The Values Behind The Results. He holds a PhD in Computer Science from MIT. Michael is a Co-Active Coach and is co-organizer of the BayALN, the agile user group in San Francisco. He loves playing, creating, and sharing games and co-organized the 2010 Agile Games Conference, the 2011 Agile Games Conference, and the 2016 Agile Games West Conference. He serves on the Scrum Alliance's Certified Team Coach (CTC) review committee and the Virtual Coaching committee. He enjoys mentoring Scrum Masters and agile coaches who want to deepen their understanding of agile. He also mentors and invests in startups through Techstars Boston.