It was a group project of staggering proportions. University of Virginia psychology professor Brian Nosek and his colleagues at the nonprofit Center for Open Science got help from over 350 scientists to repeat 100 high-profile psychology experiments published in 2008 — the largest replication study to date. In August, they announced the results in Science, forcing psychologists to face some hard truths about the reliability of their field’s studies.
The project is a response to the so-called “reproducibility crisis,” a growing concern that many published studies may not hold up to repeat testing. The problem affects other sciences, but several high-profile failures to reproduce important psychology findings have made it a particular concern.
Researchers not involved in the initial studies contacted the original authors to get feedback on their protocols; in most cases, the original researchers helped with study designs and strategies. Despite this thoroughness, while 97 of the original studies reported significant results, only 35 of the replications reported the same. And even then, the effect size (a measurement of how strong a finding is) was smaller — on average, less than half the original size. So even when Nosek’s team ended up with significant results, those results weren’t as strong as the initial reports claimed they were.
The replications are not definitive, Nosek says: “No study provides perfect evidence for or against any finding, and that also applies to the original [studies].” The replication simply adds information about the phenomenon under study, he says.
But it does mean that psychologists are starting to learn more about how to make their experiments more rigorous. “What this paper can help us do is accept that this is a problem so we can go about finding ways to fix it,” says Dorothy Bishop, a psychologist at the University of Oxford and a leading voice on psychology’s reproducibility issues. The problem won’t be solved overnight, but this study is a step in the right direction.
