The sins of analytic methodologists

We are strong proponents of improving the rigour in intelligence analysis, and have frequently advocated for the better grounding of new analysts (and in-service training) in formal logic, epistemological theory, and structured technique. At the same time, however, we are deeply mindful of the continuing unmet needs for more creative and imaginative analysis, which moves further afield from the narrowly constructed questions of current uncertainties and into the realm of the unknown unknowns.

We are thus deeply troubled to observe a frightening trend in the work of some methodologists attempting to “solve” the problems of analytic rigour through new tradecraft practices. There is an increasingly common conceit that reliance on the analyst - subject to, cognitive bias, information overload, and human fallibility – can be engineered out of the process of doing intelligence. Instead, certain methodologists would substitute organizational structures, workflow re-organization, and the introduction of supposedly superior quantitative metrics in order to create a new standard for “answers”. The underlying thrust of these efforts is to reform intelligence activities towards a more “repeatable” process, often described by industrial or scientific metaphors such “foundry” or “lab”. These typically originate from the engineering and technical intelligence disciplines, and are usually directed as criticism of typical all source efforts – particularly those grounded in social science fields or qualitative methodology. Among the recent examples of these attempts at “standardizing” the process of analysis – with new emphasis on technical and quantitative approaches – is the 2006 DNI Galileo program paper “Predictive Network Centric Intelligence.”

The fundamental flaw in many of these methodologists' efforts is that they are essentially reductionist attempts to force the difficult and oft-times messy art of intelligence entirely into the narrow box of its scientific side. While there is a place for scientific approaches, particularly in the grounding and validation of assessment, the inherently creative, non-linear, and even non-rational elements of the profession can never be completely discarded. Most recent intelligence failures have occurred, not due to a lack of precision in judgment, but from a lack of imagination in identifying, describing, and forecasting the uncertain dynamics and emerging complexities of fast-changing accounts.

Most methodologists are very good at creating shiny and intricate machines for solving puzzles to produce answers of intelligence “fact” – often with great and noteworthy expenditure of time, effort, and resources. However, the community is challenged most today by mysteries, to which there may never be a single set of answers but rather only approximations and options, set against the backdrop of an ever evolving understanding of situations obscured by the fog of war, time/distance, culture, and levels of commitment.

It is also fascinating to note that many of the methodologists reject individual level approaches to solving difficult intelligence problems, preferring instead to focus on the creation of new systems and processes. It is our contention that the analyst will always remain the heart of intelligence insight, and the things that happen inside the human mind are non-replaceable - and frequently non-reproducible – elements that cannot be dissected and pinned neatly for methodological examination. The analysis “bench” will never be deep enough to support the creation of large structures of process in which analysts become interchangeable parts. Analytic tradecraft can provide best practices to help avoid certain fallacies and other mistakes, and can demonstrate cases in which valuable lessons can be learned, but it cannot by itself produce the innovative and imaginative prediction that will help decision-makers address the kinds of challenges they increasingly face.

It is for this reason that we continue to emphasize the importance of the cultivation of the analyst above all else. We would hope that the methodologists would at least consider pursuing research, which rather than seeking to radically restructure the forms of complex intelligence activity that have evolved under brutally Darwinian pressures, will instead support the pursuit of excellence by individuals facing hard challenges in the face of serious consequences. The analytic house neither needs nor particularly wants shiny new puzzle machines – but is desperate for additional tools and techniques which can be called upon to address mysteries in times of crisis, and when other sources of inspiration have run dry.