U.S. economist Robin Hanson posed this question in the title of an article published in 1995. In it he suggested replacing the classic review process with a market-based alternative. Instead of peer review, bets could decide which projects will be supported or which scientific questions prioritized. In these so-called “prediction” markets, individuals stake “bets” on a particular result or outcome. The more people trade on the marketplace, the more precise will be the prediction of outcome, based as it is on the aggregate information of the participants. The prediction market thus serves the intellectual swarms. We know that from sport bets and election prognoses. But in science? Sounds totally crazy, but it isn’t. Just now it is making its entry into various branches of science. How does it function, and what does it have going for it?

That something like this can function in science is demonstrated in newer studies using prediction markets. For example, one about forecasting whether a study can be replicated or not, and here is how that is done: The participating scientists get 100 chips (or sometimes real money, e.g. 100$) with which they can then stake their bet on the success of the replication. They would bet more on those which they believe could be successfully replicated. Vice versa, they would stay away from those they do not trust.  From the purchase and sale of the lottery tickets a price evolves (in chips or in real currency). This price reflects the probability with which the market participants believe in the study’s replicability. This procedure was used in a recent publication. Twenty-one studies from the field of psychology published between 2010 and 20115 in Nature or Science were repeated (Colin et al.). At the same time, a group of students and scientists who did not participate in the replication and who were not even necessarily acknowledged experts in the respective fields, were able to set chips they had bought, and bet on the replicability of each of the 21 studies. The result: an almost perfect forecast of those studies which were then successfully replicated and of those which were not replicable. A classic survey served as control: Do you believe that Study X or Y will be replicable? The survey results were however not better than those that can be achieved by pure guessing. In another study a prediction market was used to predict the outcome of the so-called REF for chemistry  (Munafo et al.). The REF (Research Excellence Framework), a highly complex and expensive procedure, is used by the British government in evaluating its universities, and its funds are distributed on the basis of the results (Excellence!). A simple prediction market, with only 13 chemists from student to professor, predicted fairly accurately the results of the REF 2014 for all 33 chemistry departments in England. Could this approach have been used to chose the Clusters funded in the German Excellence Initiative? Without thousands of scientists having to write up applications instead of doing their research work; without an army of international reviewers journeying up and down the Republic?

How come betting can render better forecasts than surveys or peer review? This is nothing new. We know that from betting in sports or on election results; They reach an astounding predictive power, almost always better than do surveys. One factor for this is, I suspect, that the incentive to grapple with the object of the survey is higher when there is something to be won. And when they are only “game markets”, and do not bet real money as do some prediction market studies in science. In Germany these could not be conducted with real money, because that would be illegal gambling! A further factor could be that in a prediction market you can spread your bets over a longer time-span by selling or purchasing chips; So you can switch candidates, maybe by looking at the course value of the bet, which can change with sales and purchases of other players. More important perhaps is the swarm intelligence that sets in when many people with diverse knowledge or perspectives take on a question.

But is it all just a game… or are there serious applications in our scientific system for prediction markets? One limitation of this method is naturally that it only works out questions that can be dichotomized. Is a study replicable, or not? Is a particular result right or wrong? Should a particular question be examined or not? Does the application merit funding or not? The outcomes of the market are then probabilities, a prediction market does not feed back content. Nonetheless the example mentioned above from the field of psychology demonstrates that a prognostic market can facilitate a straightforward and apparently very precise consensus in a field of research.

Research funders could use it when deciding which research program should be prioritized and established. The decision would then be communicated to the community, and perhaps to other stakeholders (e.g. patients) as well. The question posed frequently in translational medicine is whether a substance which is effective in model systems but expensive and potentially dangerous for patients should be put into clinical development or not. A prediction market would be a simple procedure to select promising drug candidates by the scientific community. Candidates could even be sorted according to their probability of successful translation. Naturally, that does not guarantee success, but in the absence of objective criteria we currently stake our bets on the opinion of the experts, albeit not on those of the swarms, but rather on select individuals (‘experts’) who are often pursuing their own interests.

Even if prediction markets won’t make it into the mainstream, wrestling with solutions that deviate radically from ordinary practice gives us an informative look into the mirror at the status quo – with all its strengths and weaknesses. And we realize: It isn’t optimal and there’s also another way to do it!

A German version of this post has been published as part of my monthly column in the Laborjournal: http://www.laborjournal-archiv.de/epaper/LJ_18_11/20/index.html

 

References

Robin Hanson (1995) Could gambling save science? Encouraging an honest consensus, Social Epistemology: A Journal of Knowledge, Culture and Policy, 9:1, 3-33, DOI: 10.1080/02691729508578768 http://dx.doi.org/10.1080/02691729508578768

Michael Thicke (2017) Prediction Markets for Science: Is the Cure Worse than the Disease?, Social Epistemology, 31:5, 451-467, DOI: 10.1080/02691728.2017.1346720  https://doi.org/10.1080/02691728.2017.1346720

Dreber A, Pfeiffer T, Almenberg J, Isaksson S, Wilson B, Chen Y, Nosek BA, Johannesson M. (2015) Using prediction markets to estimate the reproducibility of scientific research.  Proc Natl Acad Sci U S A. Dec 15;112(50):15343-7. doi: 10.1073/pnas.1516179112. Epub 2015 Nov 9

Munafo MR, Pfeiffer T, Altmejd A, Heikensten E, Almenberg J, Bird A, Chen Y, Wilson B, Johannesson M, Dreber A. (2015) Using prediction markets to forecast research evaluations. R Soc Open Sci. Oct 28;2(10):150287. doi: 10.1098/rsos.150287. eCollection 2015 Oct. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4632515/

Colin F. Camerer, Anna Dreber, Felix Holzmeister, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Gideon Nave, Brian A. Nosek, Thomas Pfeiffer, Adam Altmejd, Nick Buttrick, Taizan Chan, Yiling Chen, Eskil Forsell, Anup Gampa, Emma Heikensten, Lily Hummer, Taisuke Imai, Siri Isaksson, Dylan Manfredi, Julia Rose, Eric-Jan Wagenmakers & Hang Wu. (2018) Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviourvolume 2, pages 637–644  https://www.nature.com/articles/s41562-018-0399-z

 

By the way: Robin Hanson suggests prediction markets for finding consensus in research which are similar in layout, radicality and innovation to that of the mysterious inventor of the blockchain, Satoshi Nakamoto. Starting from a criticism of the usual payment system, Nakamoto suggested a digital currency principle which not only lured speculators but in the meantime is being applied in a number of distributed bookkeeping practices. In his legendary article, Nakamoto suggested a digital payment system.

Nakamoto, S (2012) Bitcoin: A Peer-to-Peer Electronic Cash System. http://article.gmane.org/gmane.comp.encryption.general/12588/