It was a group project of staggering proportions. University of
Virginia psychology professor Brian Nosek and his colleagues at
the nonprofit Center for Open Science got help from over 350
scientists to repeat 100 high-profile psychology experiments published
in 2008 — the largest replication study to date. In August, they
announced the results in Science, forcing psychologists to face some hard
truths about the reliability of their field’s studies.
The project is a response to the so-called “reproducibility crisis,”
a growing concern that many published studies may not hold up
to repeat testing. The problem affects other sciences, but several
high-profile failures to reproduce important psychology findings have
made it a particular concern.
Researchers not involved in the initial studies contacted the original
authors to get feedback on their protocols; in most cases, the original
researchers helped with study designs and strategies. Despite this
thoroughness, while 97 of the original studies reported significant
results, only 35 of the replications reported the same. And even then,
the effect size (a measurement of how strong a finding is) was smaller —
on average, less than half the original size. So even when Nosek’s team
ended up with significant results, those results weren’t as strong as the
initial reports claimed they were.
The replications are not definitive, Nosek says: “No study provides
perfect evidence for or against any finding, and that also applies to the
original [studies].” The replication simply adds information about the
phenomenon under study, he says.
But it does mean that psychologists are starting to learn more about
how to make their experiments more rigorous. “What this paper can help
us do is accept that this is a problem so we can go about finding ways
to fix it,” says Dorothy Bishop, a psychologist at the University of Oxford
and a leading voice on psychology’s reproducibility issues. The problem
won’t be solved overnight, but this study is a step in the right direction.