An attempt to replicate the results of some recent neuroscience papers that claimed to find correlations between human brain structure and behavior has drawn a blank.
The new paper is by University of Amsterdam researchers Wouter Boekel and colleagues and it's in press now at
You candownload it here from the webpage of one of the authors, Eric-Jan Wagenmakers.Neuroskeptic readers will know Wagenmakers as a critic of statistical fallacies in psychology and a leadingadvocate of preregistration, which is something I never tire of promoting either. Boekel et al. attempted to replicate five different papers which, together, reported 17 distinct positive results in the form of structural brain-behavior ('SBB') correlations. An SBB correlation is an association between the size (usually) of a particular brain area and a particular behavioral trait. For instance, one of the claims was that the amount of grey matter in the amygdala is correlated with the number of Facebook friends you have. To attempt to reproduce these 17 findings, Boekel et al. took 36 students whose brains were scanned with two methods, structural MRI and DWI. The students then completed a set of questionnaires and psychological tests, identical to ones used in the five papers that were up for replication. The methods and statistical analyses were fully preregistered (back in June 2012); Boekel et al. therefore had no scope for 'fishing' for positive (or negative) results by tinkering with the methodology. So what did they find? Nothing much. None of the 17 brain-behavior correlations were significant in the replication sample. Using Bayesian statistics, Boekel et al. say that
For all of the 17 findings under scrutiny, Bayesian hypothesis tests indicated evidence in favor of the null hypothesis [i.e. that there is no correlation.] The extent of this support ranged from anecdotal (Bayes factor less than 3) to strong (Bayes factor greater than 10).
Does that mean that the original five SBB papers were wrong? Boekel et al. are noncommittal on this point, saying only that
From the above discussion, one might be tempted to conclude that the SBB correlations tested here simply may not exist. However, as previously mentioned, a single replication cannot be conclusive in terms of confirmation or refutation of a finding.
They conclude by calling for more pre-registered replications, like the one they just did
We believe that in order to establish correlations between behavior and structural properties of the brain more firmly, it is desirable for the field to replicate SBB correlations, preferably using preregistration protocols and Bayesian inference methods.
However, not everyone is convinced by Boekel et al.'s negative claims. In particular, I heard from Ryota Kanai of UCL, who was the first author on two of the five papers under scrutiny. He cites limitations of Boekel et al.'s replication attempts
1. One of the findings was clearly replicated using a method close to our original study. 2. The region-of-interest (ROI) approach adopted by the authors underestimate correlation (because of spatial uncertainty). 3. The suboptimal methods could not be corrected because it was a pre-registered study. 4. There was no stage where the methods were formally reviewed for this study. 5. The study was generally underpowered, and most Bayes factors were less than 3, indicating that the negative results were anecdotal. 6. Given the above, the main claim that none of the results were replicated sounds a bit too strong.
Kanai also says that the publication and review process has room for improvement:
I [was] one of the reviewers for this paper... Originally, I asked the Cortex action editor (Chris Chambers) to let me write a commentary on this paper so that the readers can see both sides of story at once. But now, Wagenmakers already made their paper available online and you are about to write a blog about this study. It is true that the authors sent me their pre-registration document before they started their project. But at that time, it was not clear that I had to read it critically as I would for formal reviews. This should not be a problem for future work, if we use the pre-registration mechanisms such as Registered Report in the journal Cortex. I'm still planning to write a full commentary when the paper is officially out. However, I think your blog would be a great place to discuss this topic further. I hope this is going to be a constructive discussion for the neuroimaging field.
Nonetheless, Kanai says that he supports the publication of Boekel et al. because
I recognize the importance of pre-registration in this field to tackle the issues of the small N problem, publication bias and flexible p-hacking by post hoc changes of analysis strategies. I encountered a few problems when examining Boekel et al.'s manuscript [but] I thought it would be more beneficial for the field to discuss this in a post-publication peer discussion so that we can improve the practice of pre-registered studies specially in the context of neuroimaging.
I agree with Kanai's attitude here. I think Boekel et al.'s important paper is proof that truly rigorous, preregistered science is not only possible, but publishable in a major journal. We need more of this. Boekel et al.'s negative findings are certainly concerning. That said, it would be rash to write off all of the 17 claims as disproven just yet. Many of the null results were only 'anecdotal' in terms of the Bayesian statistical evidence. Boekel et al.'s sample was relatively small, and the methodological limitations noted by Kanai, while not obviously fatal ones, aren't easily dismissed. However, like Kanai I'm confident that an open discussion is the best way forward. Preregistration itself helps facilitate this. Kanai's commendable approach is a world away from the kind of hostile, defensive reactions that, sadly, sometimes greet failures to replicate.
Boekel, W, Wagenmakers, E-J, Belay, L, Verhagen, J, Brown, S, & Forstmann, BU (2014). A purely confirmatory replication study of structural brain-behavior correlations Cortex
