In the three preceding posts (Here, Here and Here) we have pointed out pitfalls in analytical reasoning using the situation in Pakistan as case material.
Readers are entitled to ask: What is good analysis?
What follows is my perspective on what makes for good analysis. It is not original but something I was taught by a teacher I feel I was lucky to encounter.
I enrolled for a course in Decision Analysis and this is what the teacher talked about in the first class:
The most important concept to understand is that a Decision and an Outcome are two separate things.
A Good Outcome is not necessarily the result of a Good Decision.
A Bad Outcome is not necessarily the result of a Bad Decision.
How can this be so?
Because between the Decision and the Outcome there is something called Uncertainty or Randomness, something that you can never know fully in advance and over which you may have no control.
Let me illustrate this with some examples that I have invented for a South Asian audience:
Suppose a batsman picks the wrong ball to hit and makes a lousy shot but the fielder drops the catch and the ball goes over the boundary: A Bad Decision yields a Good Outcome.
Now suppose a safe single is there for the taking and the batsman sets off but slips in the middle and is run out: A Good Decision yields a Bad Outcome.
The lesson to take away is that a Decision can be Good or Bad independent of the Outcome.
A Good Decision is one where you have taken all the available information into consideration, gone over all the alternatives possible, studied their costs and benefits, and then chosen what you think is the best course of action in the circumstances.
The Outcome could still be bad – fate may intervene, an earthquake may alter the cost-benefit calculus, a coup in Russia may freeze the credit markets. As they say, ‘there’s many a slip between the cup and the lip.’ Regardless, given what you knew when you knew at the time you made the decision, it was the best if you did all that was mentioned above.
Now replace Decision with Analysis and Outcome with Prediction.
An analysis is not considered good ONLY if it predicts correctly.
An analysis is good if it does the following:
- Marshals all the available evidence that is relevant to the analysis.
- Organizes the evidence in a way that it can be assessed carefully.
- Understands the context in which the evidence is to be applied.
- Recognizes the forces and trends that are acting upon the context.
- Surveys critically the various alternatives that are likely.
- Intuits the motivations of the key actors in the situation.
Based on the above the analyst offers his or her opinion on the most likely outcome – which could still be wrong.
Note that the good analyst acts more or less like the good doctor or the good detective in the process he/she follows in reaching a diagnosis or a conclusion.
Note also that these skills require a lot of training. There may be the rare intuitive analyst born with the gift but most of the time analysis is a learned skill that requires dedicated study.
It is not for nothing that leading universities in the West have programs to train analysts who then earn their livings as professional analysts. It is because, more often than not, good analysis yields good predictions. And when it doesn’t, it is possible to go back and study what might have gone wrong improving the analytical method in the process.
It is certainly not possible to do good analysis driving by in a car, or drawing conclusions from fluffy puppies, or hoping for the best.
TheSouthAsianIdea Weblog 
Leave a Reply