Reproducibility Crisis: The Plot Thickens

A new paper from British psychologists David Shanks and colleagues will add to the growing sense of a "reproducibility crisis" in the field of psychology. The paper is called Romance, Risk, and Replication and it examines the question of whether subtle reminders of 'mating motives' (i.e. sex) can make people more willing to spend money and take risks. In 'romantic priming' experiments, participants are first 'primed' e.g. by reading a story about meeting an attractive member of the opposite sex. Then, they are asked to do an ostensibly unrelated test, e.g. being asked to say how much money they would be willing to spend on a new watch. There have been many published studies of romantic priming (43 experiments across 15 papers, according to Shanks et al.) and the vast majority have found statistically significant effects. The effect would appear to be reproducible! But in the new paper, Shanks et al. report that they tried to replicate these effects in eight experiments, with a total of over 1600 participants, and they came up with nothing. Romantic priming had no effect. So what happened? Why do the replication results differ so much from the results of the original studies? The answer is rather depressing and it lies in a graph plotted by Shanks et al. This is a funnel plot, a two-dimensional scatter plot in which each point represents one previously published study. The graph plots the effect size reported by each study against the standard error of the effect size - essentially, the precision of the results, which is mostly determined by the sample size.