The problems of prophets and jesters

The need for more predictive intelligence is one that has seen a great deal of debate over the years. The first area of argument is as always (particularly when academics are involved) the definition of what prediction actually means, in the context of intelligence as both a process and as a product. (As much as we hate arguments over definitions, occasionally they ought to be revisited as first principles in a discussion, especially when a matter may be otherwise subject to misinterpretation.)

Our preferred view on this is that predictive intelligence means bounding the space of future uncertainties within an estimative framework. Good predictive intelligence therefore are estimates (and the tradecraft used to develop such estimates) that accurately, coherently, and pragmatically provide a view of bounded uncertainties that provide actionable insights to decision-makers that correspond closely to the actual course of future events. Good predictive intelligence also addresses the potential shocks - such as Black Swan events - that may emerge in future scenarios, in much the same way that well crafted capabilities intelligence addresses linchpins and milestones.

This is by no means an uncontroversial definition. There are those that would remove the term “predictive” entirely from the lexicon of intelligence, favoring only the specific verbiage of estimative intelligence. This we believe is a fallacy – first because the term is already in common use, formally or otherwise, and without seeking to distinguish good uses of the concept from those taught by false prophets one does a great disservice to those individuals which must work through the wider body of literature – or multiple agencies’ doctrines, where the concept may be favoured. The second reason we support discussion of predictive intelligence is because many intelligence consumers have articulated the need for improvement in the area as a key objective. There is certainly a common misunderstanding by consumers regarding the nature of what can be reasonably expected from prediction within intelligence, with the consumer’s desires leaning more towards the impossibilities of fortune telling. However, this makes it all the more critical that the purpose (and limitations) of predictive intelligence be communicated effectively to prevent such misunderstandings from colouring a consumer’s perceptions of products which are crafted to the best possible (realistic) standard – especially analysts are not issued a crystal ball with which to meet unrealistic and Hollywood influenced standards.

We see no conflict with the classic view of estimative intelligence in this discussion (although in some circles, we acknowledge that we may be a distinct minority of this opinion). After all, no less a luminary than the esteemed Harold Ford wrote that the among the questions that estimative intelligence seeks to answer are “what trends seem likely for the future, and how those trends might be affected in the event certain contingent events should occur” and that “the purpose, character, and significance of these courageous estimates of future unknowns has been recognized by many observers.”(The quotes are taken from his 1993 AFIO monograph on the topic, for those keeping score.) This very clearly refers to predictive intelligence in the same fashion that we describe it.

In a way, the debate over terminology and concepts – and in reality, the underlying purpose of what intelligence should seek to be – reminds us of the same debate over whether or not intelligence professionals should be responsible for examining questions of adversary intentions. While that debate has largely been settled conclusively in favour of that purpose, it was not always so. A good deal of literature – particularly that written in the earlier Cold War military context – made many of the same kinds of arguments regarding the impossibility of divining intention as we hear made regarding the prediction of future uncertainties. (And we should note that we still occasionally hear the arguments regarding intelligence on intentions when talking with law enforcement folks or others outside of the community.)

Having spent the foregoing establishing context, we recently also encountered a post by Charles Stross, one of our favourite jesters from the futurist court, which discussed the increasing difficulties of understanding technological drivers in out-years predictive scenarios given the accelerating pace of change (and adoption of that change). The points is well made by a chart taken from the Economist, depicting the deltas of technology penetration throughout history.

While technology drivers are often overstated in many futures intelligence exercises – particularly those conducted by individuals with their own stake in a given development or industry sector – there is no denying that from the perspective of certain intelligence accounts technology is often the defining feature around which other social, political, economic, and military events develop.

And it is not merely the rate of adoption within general societies that must be considered by intelligence professionals seeking a greater level of predictive analysis. The pace of hostile innovation has also radically accelerated, particularly when it comes to adoption of new technologies that enable asymmetric engagement, and which support the resilience of non-state actors under intense selection pressures. Many of these innovations are decidedly less than high tech – but as little as a decade ago still would have been the stuff of science fiction and laughed out of the briefing room had any intelligence analyst been foresighted (and naively foolish) enough to raise them as potential issues. We would do well to ensure that our current analytic environments do not likewise encourage such a narrow minded focus that would miss the sweeping rate of change that is bearing down on us, even as ridiculous as any given manifestation sometimes may seem from our current vantage point.

This is one of the reasons we seek to encourage the jesters, and to exhort the courtiers and fops to admit a bit more levity into their dance. For somewhere in the scullery there is a hard working young analyst that listens, and nurtures their own private vision of a future that may well be more probable than any included in the official powerpoint decks. If that analyst does not come forward for fear of the reaction within his shop’s environment, or is not given the opportunity to cultivate and explore those ideas, the loss of that concept may well contain the seeds of the next failure of imagination.

Delving deeper into prediction markets

Michael Abramowicz of George Washington University has been guest blogging at Volokh Conspiracy for a short time now, and he has given us much food for thought on the topic of prediction markets. There is easily enough material for an entire book, and unsurprisingly, he has written one (that is now on our must read stack); as well as his own blog site. The Volokh post series has been:

• An Intro to Prediction Markets and the Liquidity Problem

• Prediction Markets vs. Conventional Wisdom

• A Quick-and-Dirty Empirical Defense of Prediction Markets

• Manipulation of Prediction Markets

• Deliberating with Prediction Markets

• Predicting Decisions and Their Effects

• Normative Prediction Markets

• Why Normative Markets?

• Nineteenth-century prediction markets

We will no doubt have more to say on the topic ourselves in good order. However, it is a subject that deserves deeper reflection, especially given our acknowledged skepticism of such efforts – and our general distaste for attempts seeking to create artificial numeric precision.

Nonetheless, we are quite grateful to the author for his work, which offers a unique contribution to the literature in an area of great interest to the IC. Whatever one may think of the technique, it is worth exploring with the same rigour as any new methodology.

The perils of arbitrary and false precision

We find quite unhelpful the recent academic obsessions over estimative language – largely an exercise in the introduction of a numerical system which offers a degree of apparently comforting but entirely arbitrary, and therefore utterly false, precision. It seems however that we are in one of those cycles which seem to come along in the intelligence community every few decades or so, in which the numerologists and other soothsayers attempt to reshape the profession into their own desires for a more “scientific” practice.

Let us be clear. There are times when quantitative analytic methodology is vital – but there are far more situations in which it is misapplied, misunderstood, and entirely out of place. The latter comprise the vast majority of scenarios in which analytic tradecraft is called upon – not the least of which may be attributed to the highly unbounded and indeterminate nature of the problems with which we must grapple. And any time in which a quantitative basis has not been established, the insertion of numerical percentages for predictive purposes is little more than a farcical exercise in arbitrary selection. Over time, you may attune a group sufficiently in order to calibrate its judgment of these percentages in such a way as to create a consistency within that shared hallucination. However, this does not alter the underlying fallacy upon which such a house of cards is built. This is clearly shown in the number of cases in which the naive predictor is a better estimate of potential than the much vaunted group of experts’ judgment. Thus even in finance, the most precise of arenas, built upon the foundation of values, you will find predictions expressed equally alongside hedges – and the market littered with those who have failed to impose arbitrary figures on a highly indeterminate problem.

One of the greatest challenges in intelligence analysis is to understand the limits of prediction when going about the hard business of estimation. That understanding should shape the analyst’s focus on what ought to be examined for predictive possibility. These are, properly: the scope and nature of trends, drivers, and future scenario outcomes – and not the capricious shadings of difference between mathematical expressions of probability.

Estimative language has not been expressed through probability percentages for the sixty plus years of the intelligence community’s modern incarnation for good and well contemplated reasons. While the abstraction of the clean and sterile realm of mathematics is often a welcome change from the messy and hard realities of intelligence, that abstraction too frequently is used as a shield and an intellectual refuge for those unable or unwilling to embrace the challenge of actually doing intel.

Scientisim in intelligence analysis is a particularly seductive heresy. It offers the false promise of greater insight, should only additional efforts be applied more systematically, more rigorously, or with more and better data. But it has not been given unto us to see the future – no matter how carefully we might craft our equations. We may simply chart the boundaries of its outlines, and discuss the implications within the uncertainty space so described.

We have no doubt that we will revisit this discussion in short order. For now, however, we would close with an excellent reminder of the vast gulf of differences that may be concealed with that change of a single degree of significance in numerical expression. Originally produced for IBM, this admittedly dated video still serves to explain the staggering concepts of scale in a world of large numbers. (h/t to Thoughts Illustrated for pointing out its online incarnation.)

A hard look at prediction markets

Rarely does an analytical methodology garner attention in the manner that has marked the discussion of prediction markets. From controversial origins to increasingly widespread public adoption, we think more pixels have been spilled on this single approach than on perhaps any other methodology short of ACH.

We have written about such techniques before. Perhaps we might group them more generally under the moniker of arbitrary quantitative forecasting. Arbitrary, for the numbers themselves however derived have only relative meaning in the assessment of probabilities (including even financial data, which although it carries with it information about the state of a transaction series or commodity, responds as much to the complexities of interactions between financial entities as it does to those factors of relevance for intelligence forecasting.)

It has been difficult, however, to evaluate the effectiveness of the technique amidst all of the hype. Certainly, we know of no significant influence on ordinary analytic tradecraft. The real business of intelligence continues much as it always has. This does not necessarily invalidate a methodological experiment, for there is certain room for more specialized vehicles to address unique problems or support new product lines. This is the usual fate of a new and uncertain methodology, and is not a bad thing in and of itself. (Although we do not that adoption of new methodologies has been recently accelerated, which we can attribute in part at least to the more widespread discussion within a growing literature. A new technique or approach might have lingered for decades before seeing significant use, but now may find a home – even if in a specialized shop – within months or years.)

Validation has always been the bane of methodologists. However elegant their theories, they are doomed to academic irrelevance unless adoption occurs across a sufficiently representative section of the community. And absent validation, adoption – especially in cases where significant implementation effort is required - will always chancy. In the face of production pressures and surge requirements, analysts will in almost every case fall back upon processes with which they are familiar – structured or otherwise. Prediction markets by their very nature tend to require a substantial up-front effort for highly uncertain results.

We are thus grateful to the folks at Google, along with coauthors from NBER and Dartmouth, for publishing some of the first real results of their internal prediction market. The study covers nearly three years of the operation of an exchange which handled over 70,000 transactions – each conveying a degree of belief on one of almost 300 particular questions, on behalf of 1500 active employees (although nearly 6500 held accounts that were not used.) Interesting, they identify unexpected influences due to physical proximity, as well as the impact of cognitive bias towards optimism based on employee fiscal considerations created by Google’s rising share price. Also quite interesting was their observation that new employees were more influenced by this bias, and that staff with longer tenure within the firm tended towards more calibrated judgment – a not inconsistent phenomenon within any analytic activity.

As warrant to the authors’ point regarding proximate location influences on information sharing, it was also revealed that Google employees moved offices approximately every 90 days. If ever there was a indicator of a complex and unstable system… but of course, we are aware of quite a few community elements that would meet or even exceed this frequency.

At least a third of all market questions were purely “fun” topics, while nearly half did not have direct impact to Google. This begs the question of how much of the activity was merely socialized gambling using virtual currency vice the exercise of deliberate judgment regarding the potential future environment – something that will plague almost any prediction market collaboration. While fun helps drive adoption, and play can lead to divergent insights, it is easy to envision such a mechanism as becoming a drain on the hard questions of the real topics.

All in all, the paper is well worth reading and carries with it quite a bit of food for thought to sustain those debating the utility and applications of prediction markets within the intelligence community. We admit to a growing skepticism regarding the value of the methodology that this study has only served to reinforce. Given the total time, resources, and intellectual energies required to support such an endeavor, these kinds of outcome do not in our view necessarily justify the effort. However, we remain open to the potential that such mechanisms capture effort which might otherwise be entirely undirected, and therefore may create insight where other techniques would not. These remain in our minds open questions, and worthy of further research.


h/t Marginal Revolution and Midas Oracle

Intel 3.14159265

A lot has been said recently about the application of Web 2.0 technologies to the intelligence community. The debate has also attracted new bloggers to the field – some academic / student, some more professional.

But this debate occurs at a time when many are seeking to identify the next generation of technologies beyond the current crop of lightweight / social / rich experience / web as platform entrants that have defined the generation. Some commentators have even gone so far as to declare the Web 2.0 meme dead – ironically at around the same time as the intelligence community has just begun to manage to wrap its collective head around the possibilities of the technology, with things like Intelink blogs, Intellipedia, and now the new A-Space.

We remain uncertain what the next new wave of technologies might bring to the community. However, certain tantalizing possibilities do present themselves. The Web 2.0 revolution is fundamentally a change to the way information is shared and internalized by those tasked with production – in short, the way analysis is done. (And contrary to the self-aggrandizing claims of certain university types, real distributed collaborative analytical work is being done in the environment of the community’s wikis and blogs, not merely just descriptive summation.) New analytic tradecraft is developing, enabled by these new technologies, in ways that it is frankly impossible to fully predict. We have only begun to observe the first outlines, hinting at what might eventually be the native competence of these environments.

Given that Intel 2.0 is all about exchange and analysis, the next iteration of revolutionary transformation will likely change forever the dynamics of intelligence collection. The systems and processes which dominate collection as a problem set remain firmly mired in industrial age models, part of the long legacy of the cultures which gave them birth. The new generation entering these fields will bring with them changes which cannot be forestalled for long.

Exactly what the nature of these changes might be is another question entirely, however. The community has not fully grasped the implications of iteration 2.0, and peering forward to what will come after is less an exercise in forecasting as it is in fortune-telling. In this, however, we unapologetically look to the jesters at the futurists court’s table – the speculative fiction authors, who may fearlessly explore these new spaces unbound by the constraints of the mundane.

It is from one such writer we recently observed the fascinating potential for emergence at the intersection of several technologies and social changes. Charles Stross is no stranger to writing about intelligence in fiction – quite enjoyably crossed with elements of the fantastic in an elaborate Cold War allegory (which he has also sought to explain in an essay on "The Golden Age of Spying", well worth reading even for those professionals which otherwise eschew the genre). His latest novel, Halting State, touches again upon the work, this time presenting a series of intriguing suggestions regarding future trajectories of the field. Among his concepts (one of which led to the title of this post) are that alternative reality games might be adapted to training a pool of unwitting subjects for future intelligence and related support tasks, that an age of nearly ubiquitous networks will lead to new emphasis on classic HUMINT operations, and potential radical changes in field operations will be enabled by the introduction and common adoption of augmented reality vision displays. He further highlights the nature of the potential future adversary – the “blacknet” of highly networked transaction driven hostile connectivity which enables a market of illicit goods and services (including those things of economic value in persistent virtual worlds) exchanged on behalf of criminal and other adversarial interests.

Like all good speculative storytelling, it is based on elements of the future which are already here, but not evenly distributed (in the words of Gibson). A fascinating menu of potential, to say the least, the implications of which are well worth exploring in a more formal manner within the community. Again, if ever there were a role for the intel studies academia…

Markets for prediction versus prediction from market data

Our readers will recall that we have long been fascinated by the potential for new concepts in analytic methodology which might allow the imperfect instrument of predictive intelligence to be refined and applied under new conditions. Among the vehicles which have emerged over the past few years that might seem to offer promise is the idea of the prediction market – instantiated in various forms under DARPA, Long Bets, and other iterations.

We have yet to see any truly definitive examples of such a market’s utility to the practical and very real problems of forecasting for intelligence issues. We are also increasingly mindful of the new issues which emerge during the implementation of such markets – especially those related to strategies which successfully game the market from the perspective of an individual “trader” (responding correctly to market conditions to create “wins” for his own portfolio), but which degrade the pure anticipatory value of market based information due to meta-strategies focused one the market itself rather than the issue to be predicted. In this, we recall Nassim Nicholas Taleb’s admonition regarding successful trading strategies that only require one good year in a century to produce a profit, and who insulate against major losses through a strategy of slow cuts.

We thus were initially quite interested in a new paper which attempts to evaluate the Surge through the lens of a prediction market approach. However, the key difference we note is that the paper is an attempt to measure not a market consisting of predictive judgments, but rather to derive predictive value from a market of financial judgments. In our view, the attributing causalities to the decisions of finance is a far more difficult business, and so prone to errors of bias that we would question its practical utility.

However, it is an interesting example of the difficulties of cross disciplinary applications in the intelligence space. (And a clear definition of why intelligence is an art and science worthy of study of its own, as opposed to merely a bastardized aggregation of various social sciences – as some critics have in the past charged.) We encourage such efforts, not because we hope to gain transcendent insight on specific issues through any particular experiment, but so that we can continue to incorporate new approaches into the business of intelligence through trial and error. Identifying a dead end pathway is as valuable as finding the next new thing.

To this end, we would also recommend our readers to the excellent item at Mapping Strategy regarding the perils of actuarial approaches to prediction, in which we would heartily second the call for caution in making assumptions regarding the predictability of events based on their historical occurrence. (This is one of the reasons we have been so critical not only of Schneier et al’s comments on counterterrorism issues, but also of Pape’s work regarding suicide attacks.)


h/t Marginal Revolution and Economist’s View

Predictive markets in futures studies

The concept of using markets as a predictive intelligence tool for future studies has been too often critiqued by those who do not understand the nature of conclusions which can be drawn from market data. While we may question the collective groupthink that sometimes dominates on particular issues, and often find ourselves quite contrarian to the whole, the concept is an interesting one and deserves far more attention and exploration than it has been given. The politicization of the concept in the wake of media attention a few years ago has not helped in the least.

We have seen a few new efforts in the space - not least of which in Google’s internal predictive market - and there is even a journal for the field, but we fear that too much of the potential value has been destroyed by the politics of the thing, at least for this generation.

We also find the introduction of the economic element the most critical aspect of the successful predictive market. We have long been deeply suspicious of the arbitrary introduction of quantitative values into inherently non-numeric intelligence problems, and regrettably have seen far too much of a trend in recent years (especially in certain segments of the academy) that seeks to assign a false precision to ideas which exist entirely within subtle variations of the gray, better communicated through language and narrative than dry figures attempting to establish an aura of pseudo scientific rigor. The economic element is indeed another arbitrary figure – but there is something that focuses the mind when the scales are well perceived in terms of other weights such as opportunity costs or even the simple fixed pool of limited resources.

Thus it is with interest we note that the Long Now Foundation’s efforts to create a predictive market of its own, Long Bets. The Long Now effort has produced some very interesting forecasting and think pieces looking at the truly out years perspectives on the human condition, and its efforts here look quite promising. The market itself is largely a pairwise establishment, between those offering predictions regarding future scenarios, events, and drivers and those who would challenge the statement against a fixed sum to be donated to charity by the inaccurate party. As such, it is far less a mechanism for determining the wisdom of the crowd than it is a matter of watching the alpha geeks. For currently, the participants that are backing and challenging these predictions, are a smaller elite of highly connected, typically highly interesting individuals with strong existing public records in analysis and forecasting of difficult issues in complex and dynamic problem spaces. But this is very much a thing interesting in its own right.

In a way, this reminds us of the process in use at Stratfor, as recounted by their “Chief Intelligence Officer” George Friedman, in which their analysts are forced to make predictions for an internal record (apart from published FINTEL) regarding their assigned accounts on a quarterly and annual basis. These predictions are permanently tracked and make up a part of the analyst’s performance review and professional development process. (We would be very interested to see a study of their results over time correlated with other measures of analyst performance, if only from the perspective of a kind of reputation mechanism.)

The Long Bets effort bears watching. And more importantly, this is a low overhead mechanism that can easily be duplicated within the intelligence community, perhaps even as simply as establishing a short Intellipedia page to track IC internal versions (although the key financial elements are another matter entirely…. Perhaps the “bets” can be paid in non-currency commodities of unique value only within the vault…)

Illustrating the difficulty of futures intelligence

This item by science fiction author Charles Stross is an excellent micro case study of the difficulties of doing good futures intelligence and scenario projection. The issues and concerns of the day, especially in the alternative planes of finance and thought that comprise the Parallel World, are only faintly visible in the distant Starlight. The driving passions, and serious concerns, of those caught up in the future mysteries that will be the realm of tomorrow’s intelligence rely on a series of nested assumptions and understandings that we can only barely begin to sketch the outlines of today.

In a way, our earlier admonition regarding the need to be constantly aware of the fallibility of trying to doing intelligence in the incomplete information space of OSINT also holds true for futures intelligence. It requires a great deal of personal humility, and a very open mind, to get futures intel right. The same even goes true for its easier cousin, horizon scanning, as selection and emphasis is naturally biased by the limitations of one’s own perspective, inevitably and irreducibly rooted in the now. Frequent trips to face Smoking Mirror are called for….

Warning examples for would-be futures analysts

We have recently begun following the quirky Paleo-Future blog out of the horrifying spectacle that one day our own forecasts may be featured therein - though we are comforted in that our writings will no doubt likely remain quite obscure, especially in comparison to things such as AT&T advertising or Apple's early pop culture visions. But we learned early to always be mindful of the examples of other’s mistakes, in order to seek to avoid them ourselves.

Collections such as Paleo-Future serve to point out well that most of what passes for futures analysis is merely a concise summation of the features of the present, exaggerated in a manner which reflects the interests and cognitive biases of the day. And while the future may already be here in uneven distribution, interesting parts of that future also emerges at the intersection of today’s forces and drivers that create complex higher order effects in ways that will always be difficult to predict. That’s why futures studies will remain just as much of a job as any other part of the intelligence equation.

We have been troubled by the lack of effective predictive analysis tradecraft in most intelligence analysis instruction, and a deep misunderstanding of futures studies techniques displayed on the part of many of the faculty and students emerging from typical academic programs. The process of authoring a fifteen year out-years assessment is far different than developing key judgments for an estimate regarding a current intelligence problem, but it seems too few times are those differences recognized or addressed.

This failure is not for lack of resources. The futures studies field has been producing a body of literature that frankly already nearly exceeds that of the intelligence studies academia, from a much smaller base of much younger institutions. The basic text as always remains the Art of the Long View, but there are numerous others tackling applications ranging from business strategy, technology developments, and a number of key national intelligence questions. Demonstrated project efforts abound, most notably in the United Kingdom’s multiple horizon scanning efforts, the National Intelligence Council’s excellent recurring series, and other private sector efforts. And for as much as Proteus is cited for its conclusions regarding the potential intelligence environment of 2020, the process by which it was developed is equally ignored.

This issue ties into the greater concerns regarding the lack of strategic imagination and strategic thinking within the intelligence community. It is widely acknowledged that the overwhelming press of current intelligence demands continues to rob time and effort that might otherwise have been devoted to longer term concerns, and to the kinds of interactions that lead to creativity and insight in futures problems. But it is hard to worry about tomorrow when one is always fighting fires today. Even when scarce time and analytical resources are devoted to futures studies questions, they often focus on easy, media-centric shibboleths which are politicized from the start by their very nature.

Futures studies, forecasting, and other forms of predictive analysis deserve far more attention than they current receive. There is some hope on the horizon, as some of the best in the community are shifting posture to emphasize a far more forward looking, forward leaning approach in intelligence studies – integrating predictive and opportunity analysis at every level. Hopefully, these efforts and their follow-on imitators bring new focus to futures intelligence.