Markets for prediction versus prediction from market data

Our readers will recall that we have long been fascinated by the potential for new concepts in analytic methodology which might allow the imperfect instrument of predictive intelligence to be refined and applied under new conditions. Among the vehicles which have emerged over the past few years that might seem to offer promise is the idea of the prediction market – instantiated in various forms under DARPA, Long Bets, and other iterations.

We have yet to see any truly definitive examples of such a market’s utility to the practical and very real problems of forecasting for intelligence issues. We are also increasingly mindful of the new issues which emerge during the implementation of such markets – especially those related to strategies which successfully game the market from the perspective of an individual “trader” (responding correctly to market conditions to create “wins” for his own portfolio), but which degrade the pure anticipatory value of market based information due to meta-strategies focused one the market itself rather than the issue to be predicted. In this, we recall Nassim Nicholas Taleb’s admonition regarding successful trading strategies that only require one good year in a century to produce a profit, and who insulate against major losses through a strategy of slow cuts.

We thus were initially quite interested in a new paper which attempts to evaluate the Surge through the lens of a prediction market approach. However, the key difference we note is that the paper is an attempt to measure not a market consisting of predictive judgments, but rather to derive predictive value from a market of financial judgments. In our view, the attributing causalities to the decisions of finance is a far more difficult business, and so prone to errors of bias that we would question its practical utility.

However, it is an interesting example of the difficulties of cross disciplinary applications in the intelligence space. (And a clear definition of why intelligence is an art and science worthy of study of its own, as opposed to merely a bastardized aggregation of various social sciences – as some critics have in the past charged.) We encourage such efforts, not because we hope to gain transcendent insight on specific issues through any particular experiment, but so that we can continue to incorporate new approaches into the business of intelligence through trial and error. Identifying a dead end pathway is as valuable as finding the next new thing.

To this end, we would also recommend our readers to the excellent item at Mapping Strategy regarding the perils of actuarial approaches to prediction, in which we would heartily second the call for caution in making assumptions regarding the predictability of events based on their historical occurrence. (This is one of the reasons we have been so critical not only of Schneier et al’s comments on counterterrorism issues, but also of Pape’s work regarding suicide attacks.)


h/t Marginal Revolution and Economist’s View