Brain scanning is big at the moment. In particular, the technique of functional MRI (fMRI) has become hugely popular within neuroscience. But now a group of big-name neuroimaging researchers, led by Russ Poldrack, have taken a skeptical look at the field, in a new preprint (currently under peer review) called Scanning the Horizon: Future challenges for neuroimaging research. Poldrack et al. do a great job of discussing the various problems including limited statistical power, undisclosed analytic flexibility (producing scope for p-hacking) and inflated false positive rates in the software tools used. They also cover proposed solutions including my favorite, preregistration of study designs. Neuroskeptic readers will find much of this familiar as I've covereda lot ofthese issues on this blog. The authors also offer some interesting new illustrations of the problems. I was particularly struck by the observation that out of a sample of 65 fMRI papers retreived from PubMed, 9 of the papers used FSL and SPM software for most of the data analysis but then switched to the seperate AFNI software package for the final inference step of multiple comparisons correction. There seems to be no good reason to do this. FSL and SPM provide their own multiple comparisons correction tools. Although it's impossible to be sure what's going on here, it looks like researchers may be 'shopping around' for statistical tools that happen to give them the results they want. Poldrack et al. also provide a neat graph showing the sample sizes in fMRI studies over the years. The lines show the estimated median sample size. The typical size has increased steadily from about 10, in the 1990s, up to around 25 today. This is still, in absolute terms, rather small.
One issue that's not covered in Scanning the Horizon is problems in the interpretation of fMRI results. Even if researchers use the correct statistical techniques and software, it is easy to misinterpret or overinterpret the results. One very common problem is the so-called imager's fallacy in which the existence of a statistically significant 'blob' in one area of the brain, and the absence of a blob in another area, is taken as evidence of a significant difference between those two areas.
