A hard look at prediction markets

Rarely does an analytical methodology garner attention in the manner that has marked the discussion of prediction markets. From controversial origins to increasingly widespread public adoption, we think more pixels have been spilled on this single approach than on perhaps any other methodology short of ACH.

We have written about such techniques before. Perhaps we might group them more generally under the moniker of arbitrary quantitative forecasting. Arbitrary, for the numbers themselves however derived have only relative meaning in the assessment of probabilities (including even financial data, which although it carries with it information about the state of a transaction series or commodity, responds as much to the complexities of interactions between financial entities as it does to those factors of relevance for intelligence forecasting.)

It has been difficult, however, to evaluate the effectiveness of the technique amidst all of the hype. Certainly, we know of no significant influence on ordinary analytic tradecraft. The real business of intelligence continues much as it always has. This does not necessarily invalidate a methodological experiment, for there is certain room for more specialized vehicles to address unique problems or support new product lines. This is the usual fate of a new and uncertain methodology, and is not a bad thing in and of itself. (Although we do not that adoption of new methodologies has been recently accelerated, which we can attribute in part at least to the more widespread discussion within a growing literature. A new technique or approach might have lingered for decades before seeing significant use, but now may find a home – even if in a specialized shop – within months or years.)

Validation has always been the bane of methodologists. However elegant their theories, they are doomed to academic irrelevance unless adoption occurs across a sufficiently representative section of the community. And absent validation, adoption – especially in cases where significant implementation effort is required - will always chancy. In the face of production pressures and surge requirements, analysts will in almost every case fall back upon processes with which they are familiar – structured or otherwise. Prediction markets by their very nature tend to require a substantial up-front effort for highly uncertain results.

We are thus grateful to the folks at Google, along with coauthors from NBER and Dartmouth, for publishing some of the first real results of their internal prediction market. The study covers nearly three years of the operation of an exchange which handled over 70,000 transactions – each conveying a degree of belief on one of almost 300 particular questions, on behalf of 1500 active employees (although nearly 6500 held accounts that were not used.) Interesting, they identify unexpected influences due to physical proximity, as well as the impact of cognitive bias towards optimism based on employee fiscal considerations created by Google’s rising share price. Also quite interesting was their observation that new employees were more influenced by this bias, and that staff with longer tenure within the firm tended towards more calibrated judgment – a not inconsistent phenomenon within any analytic activity.

As warrant to the authors’ point regarding proximate location influences on information sharing, it was also revealed that Google employees moved offices approximately every 90 days. If ever there was a indicator of a complex and unstable system… but of course, we are aware of quite a few community elements that would meet or even exceed this frequency.

At least a third of all market questions were purely “fun” topics, while nearly half did not have direct impact to Google. This begs the question of how much of the activity was merely socialized gambling using virtual currency vice the exercise of deliberate judgment regarding the potential future environment – something that will plague almost any prediction market collaboration. While fun helps drive adoption, and play can lead to divergent insights, it is easy to envision such a mechanism as becoming a drain on the hard questions of the real topics.

All in all, the paper is well worth reading and carries with it quite a bit of food for thought to sustain those debating the utility and applications of prediction markets within the intelligence community. We admit to a growing skepticism regarding the value of the methodology that this study has only served to reinforce. Given the total time, resources, and intellectual energies required to support such an endeavor, these kinds of outcome do not in our view necessarily justify the effort. However, we remain open to the potential that such mechanisms capture effort which might otherwise be entirely undirected, and therefore may create insight where other techniques would not. These remain in our minds open questions, and worthy of further research.


h/t Marginal Revolution and Midas Oracle