The analysis of competing hypotheses (ACH) method can be used to evaluate multiple competing hypotheses when investigating problems. The method mitigates cognitive biases that humans experience when exploring the causes of problems.
The ACH procedure is described in chapter 8: Analysis of Competing Hypotheses of the book Psychology of Intelligence Analysis by Richards J. Heuer, Jr. The Palo Alto Research Center (PARC) developed an ACH Tool which is available to the general public at no cost when used for non-commercial or educational purposes.
Frederick J. Rüegger and Jörg Bächtiger presented The Secret Service Methods for Finding and Fixing Unexplainable Distributed Screw-Ups at the OOP 2015 conference. InfoQ interviewed them about finding causes of problems, using the analysis of competing hypothesis method and how it differs from root cause analysis and the advantages that ACH can bring.
InfoQ: How do you think people are generally doing when it comes to finding the causes of problems?
Rüegger & Bächtiger: Most of us are terrible, just awful.
Finding the real cause of a problem or the real motivation behind an action just is difficult in nature. Sometimes however, it is hard because we tend to fool ourselves in a number of different ways. Often we are simply not capable of seeing vital clues in plain sight while seeing all sorts of patterns where there is none; in short, our judgment is biased and we are blissfully unaware of it.
I remember staring at something like Class.forName("java,net.FindException") or Class.forName("java,net.8indException") or Class.forName("java,net.BlindException") for quite some time … before my eyes stopped refusing to see the obvious.
InfoQ: What makes it so important to discover the real causes of a problem?
Rüegger & Bächtiger: The last example is trivial, but if you give advice about software questions involving major risk, costs or investments, then you want to be dependable. It does not really matter if you are dealing with an unexplainable defect that already happened or with a future architectural intention.
What is important is that you leave no stone unturned to document and explain your advice and that you don't oversee any of the apparently obviously correct premises that later turn out to me chimeras.
Known uncertainties rarely are dangerous, tacit assumptions turned wrong and all the things you did not even think of will get you in the end.
InfoQ: Can you describe the analysis of competing hypothesis (ACH) method?
Rüegger & Bächtiger: ACH is a structured method invented by the CIA to decide between a set of possible explanations (called hypotheses). It works by arranging these hypotheses and some knowledge and circumstantial indications (called evidence) in a matrix.
Using ACH, you try to disprove as many hypotheses as you can by judging how each piece of evidence affects the credibility of each hypothesis and finally you assess if your evidence is diagnostic enough.
ACH is only one of a multitude of "soft" methods from intelligence that can make our life as geeks much easier; it is a sort of knockout fight of hypotheses against each other on the grounds of a state of your evolving knowledge.
InfoQ: How does analysis of competing hypothesis differ from root cause analysis (five times why)?
Rüegger & Bächtiger: Root cause analysis techniques (Five times Why, Fishbone, etc.) starts from an observable effect and an immediate cause for that effect. The assumption is that behind that visible cause there is another unobserved cause that can be guessed by repeatedly asking "… and why is that ?" or " …and what do we want to achieve with that ?".
ACH does not assume any causal structure whatsoever. Quite the contrary is true: only mutually exclusive hypotheses are considered! ACH does not seek causes; it just relates your knowledge to different possible explanations.
InfoQ: How can you formulate a good hypothesis?
Rüegger & Bächtiger: A range of analytical methods such as Root cause analysis can be used in conjunction with ACH as "hypotheses generator", especially if you have some engineering documentation to navigate.
In circumstances where all you have to work with are opinions and guesses, there are better-suited tools. I found post-it sessions and focus group meetings quite good when generating ideas. Delphi has proven invaluable to gain a quick insight by "tapping the database in the experts head" while creating consensus.
Checklists of "canned hypotheses" are finally not as bad an idea as they sound.
InfoQ: My experience is that it can be difficult to find good evidence for specific causes. Do you have suggestions how to do this?
Rüegger & Bächtiger: What constitutes good evidence? As far as ACH is concerned, any piece of information can be evidence; You may measure the quality of a piece of information or establish the credibility of its source, differentiate between primary information expressly created for the analysis at hand and secondary pre-existing information and so on.
One of the strengths of ACH is that it does not matter: you may start with a set of weak pieces of circumstantial evidence and quickly discover that the information at your disposal does not help distinguish between good and bad hypotheses. This gives you already an edge over mere intuition; with ACH you know that you're stuck because your evidence is insufficient; you would otherwise be aware of a difficulty, but arguably not know what to make of it. Knowing what you do not know is a byproduct of ACH.
Moreover, if the evidence is insufficient, you can figure out what you would have to see in order to eliminate all possibilities but one. This definitely is a good step towards setting up an information collection plan. Before you turn into a secret agent and storm out on intelligence missions, please remember that information comes at a cost: deciding if a piece of knowledge is worth getting before having it can be very tricky. Often some substitute for the information you need already exists in your organization, you just do not know where, what it is, or sometimes you do not recognize its usefulness.
InfoQ: Can you mention some of the advantages of using the analysis of competing hypothesis method?
Rüegger & Bächtiger: The one big advantage of this method is that it effectively mitigates many of the bias-induced limitations humans experience when searching for unknown causes or effects.
The trick is that ACH tries to disprove as many possible explanations as it can rather than focusing on one pet-hypothesis and trying to prove it.
Sometimes you can disprove all hypotheses but one, sometimes you cannot, but at least you can rank the still valid hypotheses by likelihood and make a qualitative statement about them.
A huge positive side effect is that using ACH you end up with a nice and ordered catalog of things that did or did not work; you can construct logically sound arguments for the conclusions your and your team came to.