Evidence-Based Management Guide - Updated

The Evidence-Based Management Guide provides a framework for organizations to measure, manage, and increase the value they derive from their product delivery.

The updated version includes a new key value area called Unrealized Value and provides example measures that organizations can use to find their own measures that fit their context.

Evidence-Based Management is an approach for organizations to measure the value that they deliver to customers.

They can use measurements to gain insight into the means that they use to deliver value and use those insights to improve their ability to deliver value and their business outcomes.

The Evidence-Based Management Guide is developed by Ken Schwaber and Scrum.org and can be downloaded free of charge.

InfoQ interviewed Patricia Kong, product owner - Enterprise Solutions, Scrum.org, and Kurt Bittner, vice president of Enterprise Solutions, Scrum.org, about Evidence-Based Management, measuring value, the relationship between value and innovation, and the updates to the Evidence-Based Management Guide.

InfoQ: What is Evidence-Based Management?

Kurt Bittner: Evidence-Based Management (EBM) is a measurement framework that helps organizations to continuously improve toward their big goals so that they can achieve sustainable growth. Organizations need to systematically articulate their strategic goals and validate those goals by measuring the value those initiatives deliver using data; using evidence.

Patricia Kong: Evidence-Based Management is a simple framework, but this information will only be as good as the concrete efforts that are made to improve using the insights it exposes. By helping organizations find the right measures to understand both the value they deliver and to improve their means of value delivery, it helps organizations cut through the confusion about measures and governance that often surrounds agile initiatives.

Earlier InfoQ interviewed Gunther Verheyen about evidence based software management, where he explained why we should manage software development based on evidence:

Our society is becoming increasingly dependent on software products and services. Software development delivers products that are often critical, to society, and for the survival of its producing organizations. The act of managing such critical activity should not only thrive on these elements alone.

The act of managing software (I prefer the activity over the role or title) should rely on more profound insights. Hence, the practice of using ‘evidence’ for managerial decisions over software. The use of such evidence does not rule out experimentation, intuition and experience, but rather complements and completes it.

InfoQ: How can organizations measure the value that they deliver?

Kong: EBM defines four Key Value Areas (KVAs):

• Current Value (CV), the value that your product delivers to your customers, today;
• Time-to-Market (T2M), the time it takes you to deliver new or improved value to customers;
• Ability-to-Innovate (A2I), which measures how effective are your efforts to deliver value;
• Unrealized Value (UV), which measures the difference between what you deliver today and the value that could be realized if your product fully addressed all the needs of all potential customers.

We think that discussions of value, and even an organization’s agility, need to consider all four of these KVAs to guide improvement.

Bittner: Most organizations need to start by looking at the value they deliver today, or Current Value. Organizations often use revenue to measure this, and if you can measure it instantaneously, it’s not a bad measure; for example, if you are selling items online, by knowing daily sales, or even moment-to moment sales, it can give an organization some sense of the value that customers experience. A better measure is actual customer satisfaction, since sometimes people buy things they never end up using, or buy things only because they have no better alternatives. Measures like Net Promoter Score (NPS), if measured as close to the actual experience as possible, can give a better indicator of value. Even measures that simply show how often a feature is used, and for how long, can give a better picture of what customers value, than does revenue.

Going deeper, measures that give insight into why the customer is using the product are even better and can serve as true aligning measures of success. For example, we’ve worked with a company that helps organizations process their insurance claims. They look at how long it takes their customers to get paid for the services they have rendered, or how frequently a claim is refused. This information helps them understand the value they are currently delivering to their customers or their Current Value. The most important measures of value delivered are found by looking at the world through the customer’s eyes.

Kong: I recently held an EBM workshop at Net Health, a healthcare software company.  After the workshop, the Net Health team went through Current Value, Time to Market, and Ability to Innovate for all of their products. They discussed which KVMs they could visualize and/or start tracking today and jumped in, starting with Unrealized Value.

Net Health uses Confluence and Tableau to visualize their data on a public page. At a minimum, these visuals are updated once a month. Many use this data personally, but each month a group of stakeholders hold a meeting and use the visuals. They use this time to discuss the areas that they need to improve, and discuss adding and tracking other data that may be beneficial.

Simply making these data points visible and transparent has gone a long way in making the organization truly aware of how agile they are. Making their Product NPS Scores transparent has allowed teams to rally around their work. Employee satisfaction, collected via a few data points, specifically is something they’ve used extensively. They were getting feedback about getting more time to focus, and so they implemented an experiment called “No Meeting Monday”. For three weeks no one was allowed to schedule a meeting on Mondays (outside of Daily Scrum). They then compared data from before, during, and after. The data looked so good that they’re trying the experiment again, but this time with two "no meeting days". They’ll measure and inspect again and repeat. They plan on using EBM more and more in 2019 to continue increasing the agility of the organization.

We also recently worked with a real estate software company who used EBM as they were adopting Scrum. Once they used EBM for one year, the company had its largest revenue growth in 10 years, with a 92% gain in adjusted EBITDA and an 85% gain in EBITDA margin. Additionally, they saw an increase in client satisfaction, and grew their employee Net Promoter Score from 26 to the high 60s. These positive outcomes are a result of hard work of the organization and their ability to use EBM to help them with transparency, self-organization, and empiricism.

InfoQ: What’s the relationship between value and innovation?

Bittner: The need for innovation is driven by a gap between what customers experience today, and what they could possibly experience in the future. Current Value is important, but it won’t tell you whether you have further opportunities to deliver more value because some of the desired outcomes of your customers are not yet met.

To determine whether there is a gap between the customer’s current experience and their desired experience, we have introduced a new KVA called Unrealized Value. Unrealized Value could exist because your products do not yet meet all the needs of your customers. Unrealized Value could also be caused by poor market penetration - you might have a great product that your existing customers love, but many other people who might benefit from it don’t know about it yet.

Kong: One way to apply EBM is at the product level. I coached a group of product owners at a global company in the transportation industry on applying EBM by first ensuring the vision of their product is well communicated within the teams. They work together and with users to understand the desired outcomes their customers have for that product (Current Value). As the product owners refine their Product Backlogs, they first re-evaluate if and how potential new features might satisfy the unmet needs of their customer and their desired outcomes. Then, when the feature is released, they measure how it impacted these outcomes. This process helps them measure Unrealized Value at the product level.

Bittner: When organizations look at how to improve the value they deliver, sometimes improvement ideas come from unexpected areas. I worked with a large telecommunications company who wanted to improve their Time-to-Market for a digital service they offered. They had an idea that reducing the time it took to build and test the product would really help them, so they wanted to look at automating these processes. On the face of it, that sounded like a good idea, but when we did a value stream analysis of where they spent their time, we found that the real source of delay was the time it took for people to make decisions. By focusing on reducing hand-offs and wait time, they were able to have a much greater impact than they would have through build and test automation.

InfoQ: What made you decide to update the Evidence-Based Management Guide?

Kong: There were two things: the first, coming from some organizations using EBM, was that while they liked the KVAs in the framework, they found they were using different measures in those areas than the ones proposed in the prior release. We needed more flexibility to let organizations define measures that make sense for them.

The second thing we had learned is that organizations needed measures to help them decide where to invest in innovation. This insight led us to introduce the Unrealized Value KVA. Measures in this area help organizations to decide whether they should keep investing on improving a product, or if they should invest in other products or services with larger satisfaction gaps between current and desired (or potential) outcomes.

InfoQ: What has been changed in the September 2018 guide?

Bittner: In addition to introducing the Unrealized Value KVA, by moving the specific measures to an appendix and referring to them as example measures, we have recognized more explicitly that organizations need to find their own measures that fit their context. Every organization, and every customer, may measure value differently. The KVAs provide a framework to help organizations reflect on the measures that make most sense for them.

InfoQ: What’s your advice to organizations that want to start with Evidence-Based Management?

Kong: First, they need to understand why they are trying to measure, and what they are trying to improve. Measures can become weaponized and used as a way to control and punish; they need to be used to improve transparency and collaboration, and to focus on what is important. That’s why we continuously differentiate between outcomes and outputs. Second, they need to create a shared understanding of what is important, throughout all levels of the organization, about what value means for themselves as well as for customers.  Communication in the spirit of transparency helps create alignment, focus and ownership.

Bittner: A good place to begin is to start measuring value, specifically the Current Value that their products or services deliver to customers. Many of the organizations with whom we have worked have only a vague idea of the value that their customers actually experience. In order to measure the value they deliver, however, organizations may have to improve their Time to Market, in order to measure value delivered more frequently, and in order to do that they may need to improve their Ability to Innovate. However, it all starts with measuring value delivered.

About the Interviewees

Kurt Bittner has more than 30 years of experience delivering working software in short, feedback-driven cycles. He has helped  a wide variety of organizations adopt agile software delivery practices, including large banking, insurance, manufacturing, and retail organizations, as well as large government agencies. He has worked for or with large software delivery organizations including Oracle, HP, IBM, and Microsoft, and is a former technology industry analyst with Forrester Research.

Patricia Kong is the Product Owner of the Scrum.org enterprise solutions program which includes the Nexus Framework, Evidence-Based Management, Scrum Studio and Scrum Development Kit. She also created and launched the Scrum.org Partners in Principle Program. Patricia is a people advocate and fascinated by organizational behavior and misbehaviors. She emerged through the financial services industry and has led product development, product management and marketing for several early stage companies in the US and Europe.