In this podcast Shane Hastie, Lead Editor for Culture & Methods, spoke to Conal Scanlon about his talk at QCon New York on Monte Carlo Simulation for forecasting feature development.
Key Takeaways
- Knowledge work is inherently variable, and estimates are inevitably incorrect
- Monte Carlo simulation projects likely completion based on past history rather than future guesses
- A small set of real data points is extrapolated to 1000 samples and that is used to produce a probability curve
- A forecast is a point in time situation – as teams get better at delivery their predictability should improve
- Everything in the delivery process should be subject to change as it is continuously improved
Subscribe on:
- 0:30 Introductions
- 1:00 The product manager role as an interface between engineering and customers
- 1:15 Knowledge work is inherently variable, and estimates are inevitably incorrect
- 1:45 Monte Carlo simulation projects likely completion based on past history rather than future guesses
- 2:05 The five steps of Monte Carlo simulation
- 2:12 Gather past data – Cycle time for a feature
- 2:33 Extrapolating from a small set of real data into 1000 different samples
- 2:57 The statistical technique of bootstrapping
- 3:20 Once the sample set is available use the size of the backlog to aggregate how long individual items will take to complete
- 3:37 Using medians and confidence levels to forecast likely completion times
- 4:00 How this differs from just counting the number of items in the backlog
- 4:15 Building in the variability that is inherent in knowledge work
- 4:45 Developer estimates do not corelate well to reality
- 5:44 Explaining how the small set of real data is extrapolated to 1000 samples and how that produces a smoother probability curve
- 7:25 This doesn’t need a statistician to accomplish the maths – it’s relatively straightforward
- 7:40 Explaining how to accomplish this on your own data
- 8:05 The need for clarity about what is being forecasted – what is your definition of cycle time
- 9:14 Using cumulative probability distribution charts to convey the uncertainty in a forecast to stakeholders
- 9:54 Using the chart to identify dates you are comfortable publishing internally and externally
- 10:52 This approach overcomes the resistance and fear of giving estimates as it is forecasting based on trends using historic data
- 11:08 Using story mapping to identify the relative complexity of features
- 11:42 Describing the story mapping technique and how conveys the complexity of a feature
- 12:48 The need to educate and socialise the approach and that the numbers are probabilities and forecasts not guaranteed delivery dates
- 13:20 Ideas from the book How to Measure Everything
- 13:40 Techniques to convey uncertainty and the fallibility of estimation
- 14:55 Using the forecasts and real data to examine the effectiveness of your development process and the underlying assumptions that are being made
- 15:25 The Product Manager is responsible for the clarity of external communication from the team about the forecasts
- 16:05 A forecast is a point in time situation – as teams get better at delivery their predictability should improve
- 16:35 Everything in the delivery process should be subject to change as it is continuously improved
Mentioned:
