Evaluating judgment
One of the most difficult aspects of the art and science of intelligence is evaluating judgment. To weigh the accuracy of any given forecast requires review after the fact, and in this field there is rarely the closure that allows one to go back to the historical record and call a deterministic score. Doing so in any timely fashion is ever more suspect – after all, the definitive review of Pearl Harbor was not completed until more than 20 years later.
However imperfectly conducted, the task is still a necessary one – particularly when the lessons learned are vital to the conduct of the ongoing Long War. There are dangers in internalizing the wrong lessons, and in fixing too much any particular mental model. But there is equally danger in a lack of self-reflection.
The ongoing press of current operations too often robs us of the chances for such reflection. And what after action analysis does occur is far too often driven by politicization or by attempts at justification, neither of which are at helpful to the practice of the profession.
It is thus pleasant to find occasionally an author whose work on epistemological matters transcends particular political eras and the crisis of the day. Dr. Tetlock is one such writer and researcher, whose work is a must read for all intelligence professionals. He spoke recently at the Long Now Foundation, which has a podcast available (for those of us who don’t have pre-filled iPods handed to us for the commute), even if their summary does not quite reflect the true nature of the discussion (being more of a pitch than an abstract).
Among the more interesting points Dr. Tetlock makes is that good analytical judgment is directly at odds with expressions of confidence. While it is a common maxim as of late to attempt to reduce the caveats normally associated with the language of prediction, his research shows that this is a dangerous fallacy. Writing for intelligence is not Hemmingway’s fiction – complex concepts must be expressed in nuanced terms, and the analyst will do his consumers a disservice if the shades of gray within these issues in favour of a false certainty to ease communication.
He also cites an interesting point regarding the true manner in which the human mind considers estimative probabilities. While the recent trend has been for many researchers and academics to push to greater ranges of strict quantitative expression, the statistical importance of such subtle distinctions are lost in the verbal and quantitative minds of the consumers. There is a good reason why levels of probability and confidence are expressed as they are in the community, and those who would seek to impose change in favour of their own pet theories should be most cautious when doing so. (For one of the better explanations in the open literature, see the declassified key findings from the July 2007 National Intelligence Estimate, which include the section “What We Mean When We Say: An Explanation of Estimative Language.)
However imperfectly conducted, the task is still a necessary one – particularly when the lessons learned are vital to the conduct of the ongoing Long War. There are dangers in internalizing the wrong lessons, and in fixing too much any particular mental model. But there is equally danger in a lack of self-reflection.
The ongoing press of current operations too often robs us of the chances for such reflection. And what after action analysis does occur is far too often driven by politicization or by attempts at justification, neither of which are at helpful to the practice of the profession.
It is thus pleasant to find occasionally an author whose work on epistemological matters transcends particular political eras and the crisis of the day. Dr. Tetlock is one such writer and researcher, whose work is a must read for all intelligence professionals. He spoke recently at the Long Now Foundation, which has a podcast available (for those of us who don’t have pre-filled iPods handed to us for the commute), even if their summary does not quite reflect the true nature of the discussion (being more of a pitch than an abstract).
Among the more interesting points Dr. Tetlock makes is that good analytical judgment is directly at odds with expressions of confidence. While it is a common maxim as of late to attempt to reduce the caveats normally associated with the language of prediction, his research shows that this is a dangerous fallacy. Writing for intelligence is not Hemmingway’s fiction – complex concepts must be expressed in nuanced terms, and the analyst will do his consumers a disservice if the shades of gray within these issues in favour of a false certainty to ease communication.
He also cites an interesting point regarding the true manner in which the human mind considers estimative probabilities. While the recent trend has been for many researchers and academics to push to greater ranges of strict quantitative expression, the statistical importance of such subtle distinctions are lost in the verbal and quantitative minds of the consumers. There is a good reason why levels of probability and confidence are expressed as they are in the community, and those who would seek to impose change in favour of their own pet theories should be most cautious when doing so. (For one of the better explanations in the open literature, see the declassified key findings from the July 2007 National Intelligence Estimate, which include the section “What We Mean When We Say: An Explanation of Estimative Language.)
Labels: analytic tradecraft, intelligence writing, psychology of intelligence, teaching intelligence
posted by Kent's Imperative at 17:19
<< Home