In December last year, researchers Brian Dias and Kerry Ressler made a splash with a paper seeming to show that memories can be inherited. This article, published in Nature Neuroscience, reported that if adult mice are taught to be afraid of a particular smell, then their children will also fear it. Which is pretty wild. Epigenetics was proposed as the mechanism. Now, however, psychologist Gregory Francis says that the data Dias and Ressler published are just too good to be true: Too much success for recent groundbreaking epigenetic experiments.
Francis notes that the Dias and Ressler paper reported many individual experiments on mice behavior and each one found statistically significant evidence of inherited fear. However,
The probability of a set of 10 behavioral experiments like these all succeeding is the product of the probabilities: 0.023. This value is an estimate of the reproducibility of the statistical outcomes for these behavioral studies.
That 0.023 means that even if the epigenetic memory effect is real and just like the paper claims it is, the probability of getting uniformly positive results is 2.3%. Francis later writes that, at if you also consider the neuroanatomical evidence also presented in the paper (likewise all positive, he claims), the probability drops to 0.004 or 0.4%.
How could the findings of Dias and Ressler (2014) have been so positive with such low odds of success? Perhaps there were unreported experiments that did not agree with the theoretical claims; perhaps the experiments were run in a way that improperly inflated the success and type I error rates, which would render the statistical inferences invalid. Researchers can unintentionally introduce these problems with seemingly minor choices in data collection, data analysis, and result interpretation.
Dias and Ressler aren't going down without a fight, however. In their
, they say that they did include some negative results, in the Supplementary Material, and accuse Francis of ignoring these. They also protest that:
We wholeheartedly disagree with the shadow that Francis... casts on our experimental design and data analysis. All experiments conducted were reported in the article, which means that no experimental data were excluded. When one conducts transgenerational studies that are dependent on the vagaries of breeding and husbandry, one uses all that are given and is never wasteful... we stand by our results as robust, reproducible, and verified by blinded assessment.
I'm not sure what to make of this. Francis's method is a good one. If there were a paper whose results were (say) so perfect that they were one in a million, that would make me seriously concerned. But it's all a matter of degree. In a case like this, with a probability of about 1 in 300, then I don't know what to think. Some people are worried that, for all we know, Francis might be calculating these probabilities for lots of papers and only reporting the lowest ones - which might be chance findings. After all, this isn't the first time Francis has made the 'too much success' argument. In the past two years he's written five articles much like the latest one (1,2,3,4,5), taking aim at various psychology papers. All of these analyses have revealed low probabilities. Francis has never (AFAIK) announced that a given paper is not suspect. That's 6/6 "positive results". Too good to be true?
Francis G (2014). Too much success for recent groundbreaking epigenetic experiments. Genetics, 198 (2), 449-51 PMID: 25316784
