Larry Maccherone, Director of Analytics and Research at AgileCraft and frequent speaker at agile conferences like QCon, gave a webinar in which he discusses the major risks and challenges when introducing metrics in an agile environment. Risks are referred as "dragons" and the techniques to get rid of them as "slayers".
The "Eight Dragons of Agile Measurement" of Larry Maccherone are the following:
1. Using measurement as a lever to directly drive behavior
Metrics should be used to show performance results, to foster self-improvement, to act as a mirror of team's behavior. Regardless specific realities, the metrics shouldn't be used to finger pointing or to influence behaviors.
2. Unbalanced metrics
Usage of metrics should follow the principle of diversity, choosing indicators from different areas. Maccherone suggests that one may select at least one metric from each of the following quadrants:
. Do it fast
. Do it on time
. Do it right
. Keep doing it
3. Metrics can replace thinking
Metrics are at service of the team to be a quantitative complement to all of the qualitative data that exists on the team. Spirit or room temperature cannot be measured using any metric. Maccherone quotes Einstein to mention that "Not everyhing that counts can be counted"
4. Expensive metrics
Advice here is to not spend too much effort to create or get metrics. Team should be concentrated on doing the best job they can so, they must be concentrated on that. Most of the metrics should derived from existing tools and records.
5. Using a convenient metric
It is mandatory to use impartial metrics. Convenient ones may lead to misunderstandings and wrong decisions. Maccherone mentioned Einstein again for the second part of the quotation: "…,and not everything that can be counted, counts."
6. Bad Analysis
He stresses the importance of working on probability scenarios that may produce various outcomes rather than making an analysis with a single outcome.
7. Single outcome forecasts
As for the previous, Maccherone suggests the use of simulation techniques like Monte-Carlo. Having results shown on normal distribution charts, different scenarios may be addressed and the nature of the conversation may shift.
8. Human emotion and bias
Maccherone's advice is to focus the discussion on the "what about is right". This way, he suggests that the decisions taken are “more free" of human emotion and bias.
In 2009, he started his efforts to reintroduce metrics in the software development industry. At that time, the paradigm was shifting to agile and the change agents were considering metrics as harmful to achieve a smooth transition to agile.
The webinar was recorded and is available for those who might be interested.
