Forty per cent of economic experiments cannot be replicated


MAASTRICHT. Last year, it was the psychologists who took a beating, now it seems that it is the experimental economists’ turn: a study by the journal Science shows that 40 per cent of the experiments have different results when repeated. According to UM professor Arno Riedl, it is not as clear-cut as that.

Just like psychologists, economists have been setting up experiments to study ‘economic’ behaviour, in particular since the nineteen-nineties. They often do so using game situations, in which students have to make specific choices or decisions that can earn them money. 

A large international research team repeated eighteen experiments. These were all published between 2011 and 2014 in two leading journals, American Economic Review and Quarterly Journal of Economics. Of the eighteen studies, seven could not be replicated, Science reports. The other eleven, on average, showed weaker results than the original studies.

"If you read the article properly" says Maastricht professor of Public Economics, Arno Riedl, who organises many game theory experiments himself, "it turns out that between 11 and 39 per cent cannot be replicated. It depends on the method of measuring that is chosen." Besides, strictly speaking, a replication study is also just a study. "To convincingly show the robustness of an experiment, one would ideally need a series of replications.”

So, for Riedl the Science article was not bad news. It actually made him happy, he says (although he later calls this slightly exaggerated). At least the result was better than for the psychologists, where 60 per cent of the studies could not be replicated.

Riedl: “That is because economists use different standards. Afterwards, they provide complete openness about the instructions and the setup of their experiments. Moreover, they contain less noise because economists do not pose hypothetical questions to their test subjects, but give them real incentives with real consequences for their own wallets, but also for other test subjects.”

That doesn’t take away from the fact that economists should replicate more studies. An important step, Riedl feels, is that the international society of experimental economists has created a journal purely for replication studies. The disadvantage, however, is that these studies cost a lot of time and money. “A researcher at SBE doesn’t get any money to collect primary data.”

Another thing that would help is setting higher standards to significance and using larger samples. The latter is also expensive. “I have just applied to NWO for a research talent grant to carry out experiments, but a mere ten thousand euro is allocated to research, while I need at least twice that amount. Funding of neuroeconomical experiments falls completely short for large samples. A single observation in a scanner costs at least five hundred euro.”

Forty per cent of economic experiments cannot be replicated