Q&A on the Book Evidence-Based Management

The book Evidence-Based Management by Eric Barends and Denise Rousseau explores how to acquire evidence, appraise the quality of the data, apply it in your management decisions, and assess the impact of your decisions. 

InfoQ readers can download Chapter 1 of Evidence-based Management.

InfoQ interviewed Rousseau about how evidence-based management differs from other management approaches, how we can differentiate between reliable evidence and things that are not evidence, how to base decisions on evidence, how to acquire and apply scientific evidence, what can be done to determine the quality of organizational data and evidence, how to find out if the evidence that we found is applicable to a decision that we want to make, and how we can improve our evidence-based management skills.

InfoQ: Why did you write this book?

Denise Rousseau: After 15 years of working to promote evidence-based management (EBmgt), we have learned what the critical skills are that make for effective evidence-based practice.  This book breaks out the six critical steps in evidence-based practice (from ask to assess) and how to proceed in thinking through the problem you are trying to address and how the sources of evidence help you do that. EB practitioners in medicine and management have taught us a lot about the critical skills needed and this book is the product of what we have learned.  Working with the learning engineers at Carnegie Mellon we have been able to lay out the principles, skills, and practices of EBmgt in a user-friendly form that makes learning straightforward.  

InfoQ: For whom is it intended?

Rousseau: The book is intended for practicing managers, educators and students as well as scholars who are conducting research on the competencies involved in making good decisions.

InfoQ: What is evidence-based management?

Rousseau: Evidence-based management means conscientiously using four sources of evidence in making management decisions: findings from scientific research, reliable and valid organizational data, representative information from stakeholders, and thoughtfully applied personal judgment and expertise. 

InfoQ: How does evidence-based management differ from other management approaches?

Rousseau: It is a career and not a course! By career I mean that an evidence-based manager keeps learning and developing skills and insights over time--practicing, experimenting, seeking new information and critically evaluating what works and what doesn’t. It is a systematic approach to making decisions and developing one’s self over time. 

InfoQ: How can we differentiate between reliable evidence and things that are not evidence?

Rousseau: All claims, beliefs and ‘facts’ need to be evaluated for their quality.  Our book provides evidence-based frameworks for judging the quality of each kind of evidence.  For example, for scientific evidence regarding what works in practice, quality criteria include whether meta-analytic results are known, and whether controls and comparisons groups are used.  For professional judgment or expertise, we look to the length of experience in specific domain where similar decisions are repeated and the opportunity for feedback regarding the results of those decisions.  If we are dealing with experiences that are limited or lack depth and feedback, that individual might not really be so expert. The book details how to evaluate the quality of these kinds of evidence along with evidence from stakeholders and organizational data.

InfoQ: Managers often need to respond to situations and be able to make fast decisions. How can they do that while basing their decision on evidence?

Rousseau: Priming the pump. The best evidence-based managers are well-read, have developed their skills and knowledge by long-term investment in understanding the scientific research related to their practice area, and spend time gathering valid organizational data to inform their decisions. When they need to make a decision, they have quality evidence in hand. My observation has been that EBMers have a lot of good information in their heads and in their networks because they build ties with subject matter experts related to their practice.  Doing so means you have a good idea of what kind of expertise and experience is needed to have quality insights into, let’s say, a financial decision, an operational problem or a change management issue. A well-informed EBMer will have subject matter experts related the problems he or she regularly faces, but is also good at accessing “go-to people” when new problems come up. That person might have a colleague who is a business librarian or a professor, like me!

InfoQ: How does evidence-based management look in practice?

Rousseau: So let’s take an example from an organization CEBMa has worked with. It is a bank operating in many countries whose back offices for market transactions are in third world locations. A new manager joined and started raising questions about whether the bank had a good risk management and a sound safety culture in the aftermath of the 2008 financial meltdown.  Some of the leadership wanted to run off and start improving the organization’s safety culture.  But an evidence-based manager would first ask, “what is the evidence we have a problem with our risk management and our trades?”  The first step in EBM is to ASK. 

The organizational data were pretty limited, but the next step is to ACQUIRE evidence. An evidence-based manager would seek out organizational data to assess whether there was evidence of a problem. It turned out that some organizational data did exist regarding trading activities where errors had been found that involved $20,000 USD or more. The evidence-based manager might also ask experts in the bank whether they thought there were issues of risk management, trading errors, etc.  Those experts indicated it was possible, especially since so little data had been historically collected, so problems might be overlooked. 

APPRAISING is the next step.  What is the quality of the organizational data and the expert judgment we have about the bank’s potential problems?  Turns out that the data were confined only to the back offices in the third world countries; most other risk-related activities weren’t well documented. (Another problem is being identified in this case).  However, the volume of trading errors did appear to be increasing in the back offices, though it was unclear if that volume was due to growth in sheer number of grades.  

The next step is APPLY where the banking leaders began looking for possible solutions to this potential problem of trading errors. After talking with experts and reviewing the scientific literature, they identified that a critical source of back office errors in many industries were people problems, particularly communication quality and interruptions.  Discussions with managers and stakeholders (people working in the back offices) lead to the identification of a set of interventions that would reduce interruptions and noise levels in the back offices where trades were confirmed, posted and financial transactions completed.  Far from deciding a complex initiative like “safety culture”, change should be implemented. The bank opted for the more easily implemented and evidence-based reduction in interruptions as a means of dealing with trading errors. This change entailed creating quiet zones in the offices for trading work and an etiquette for reducing human interruptions of worker concentration. The bank is now in the ASSESS stage, where they are measuring changes over time following the implementation of these interruption reducing interventions.

InfoQ: What makes it so hard to develop reliable evidence for management decisions?

Rousseau: It really isn’t that hard to develop reliable evidence. Most managers are over-reliant on their own experiences and don't think to question their assumptions.  They spend a lot of time solving the wrong problems and discounting good evidence accessible to them from their stakeholders and their organization. If you think of the example I gave above about the bank, one of the big issues is the bank has “error data” that is a mere count of how many BIG mistakes they make, without any attention to the error rate, that is the number of errors over the transaction volume in a given office.  If they aren’t even measuring error rate in a valid and reliable way, it is very difficult to develop reliable evidence for your decisions.

InfoQ: How can we acquire and apply scientific evidence?

Rousseau: Guided reading and maintaining good contacts with academics, educators, and other evidence-based practitioners.  Most of us rely on the pointer information we get from knowledgeable colleagues.  Build your network and then learn to search research databases yourself (google scholar is actually quite useful) when new questions come up-- and colleagues can help us refine our questions to make our searches more efficient. I always suggest checking with others and picking their brains in order to make questions clearer.

InfoQ: What can be done to determine the quality of organizational data and evidence? What's your advice on improving quality?

Rousseau: The most important issue in organizational data quality is whether you have the data you need to test whether your beliefs about the organization are really true. So if I believe my organization has a reliable backoffice in terms of transactions, do I have the data that show how many errors are made a day or a month for a given volume of transactions.? Counts tell us almost nothing; we need rates, like errors/daily volume.  If I am relying on my impressions, I am talking to myself. I live this in my own life as I measure my students’ learning in my classes before and after they take the class (and I am not talking about test scores, but actual performance on practical problems before and after they complete the course). I have learned not to assume, but that data help you identify where to put your efforts and attention.

InfoQ: How can we find out if the evidence that we found is applicable to a decision that we want to make?

Rousseau: Remember that there are four sources of evidence and all of them matter.  Like the example of the bank I gave above, they used scientific evidence to look at common causes of errors (i.e, interruptions from other people), organizational data to spot possible risks (i.e., trading transaction mistakes), expert opinions (i.e., what experienced managers thought was a likely problem and a good solution), and stakeholder opinions (i.e. concerns of people working in the back office environment.).  It is important to use MULTIPLE sources and keep in mind that different sources may help you with different parts of the decision.

InfoQ: What kind of things do we need to take into account when using evidence in our decision-making process?

Rousseau: Although there are many pertinent factors experienced EBMers pay attention to, I want to emphasize one factor all managers can easily take account of-- the stakeholders in the decisions you are making.  Stakeholders are those people who are affected by the decision, short-term or long-term, and those who influence how it is implemented.  Good scientific evidence indicates that the earlier stakeholders are involved in a decision process, the better the quality of the ultimate decision.  

InfoQ: What's your advice for assessing the impact of decisions?

Rousseau: EBM 101: Assess results!  Did the decision lead to results you desired? Were there unexpected consequences, positive and negative?

InfoQ: How can we improve our evidence-based management skills?

Rousseau: Practice and keep in mind that perfect is the enemy of the good.  The more you work on asking questions and seeking out multiple sources of evidence, the easier it will be and the better you will be at it. Get started!

About the Book Authors

Denise M. Rousseau is an H.J. Heinz II University professor of organizational behavior. She is a scholar and educator in positive organizational practices. She co-founded the Center for Evidence-Based Management and is president of its Academic Board. 

Eric Barends (Ph.D., VU University Amsterdam) is the managing director of the Center for Evidence-Based Management. He has twenty years of management experience, fifteen years at the senior management level, including five years as an executive. He advises and coaches managers, senior leaders, and executive boards of large and medium-sized companies and non-profit organizations on evidence-based decision-making. In addition, he frequently runs training courses on this topic and serves as a visiting lecturer at several universities and business schools such as Carnegie Mellon University, New York University and Australian National University.