The validity of some 40,000 fMRI research studies has been put in doubt by a new study that questions the methods used to analyze data from the brain-scanning machines. A meta-study, from the Linköping University in Sweden and Warwick University in the U.K. looked at the software most commonly-used used to process the results from these machines, and found that it produced a false-positive rate of up to 70%. “In theory,” write the authors, “we should find 5% false positives.”
You’ve read plenty of studies where parts of your brain “light up when…” I’ve written about them plenty of times. Now, the results of all those studies has been called into question. The problem isn’t the fMRI (functional Magnetic Resonance Imaging) machine itself. It’s the software used to take the raw machine scans and turn them into something we can use.
“Since its beginning more than 20 years ago, fMRI has become a popular tool for understanding the human brain, with some 40,000 published papers according to PubMed,” says the paper. “Despite the popularity of fMRI as a tool for studying brain function, the statistical methods used have rarely been validated using real data.”
Now, the team from Sweden and the U.K. have attempted that validation, thanks to today’s “international data-sharing initiatives in the neuroimaging field.” By using only data from the healthy control groups of existing studies, the researchers could expect the results to match. That is, the scans of the brains of healthy control subjects should, statistically, be the same across all studies. But the researchers found that they weren’t, and that this is the fault of the software. It’s a little like taking the same music track from a CD and encoding it into an MP3 over and over. You should end up with a bunch of identical MP3s, but in this case, you end up with a whole lot of different ones.
The software works by chopping the continuous scan into voxels, or 3D pixels. It then analyzes these voxels to look for clusters, using statistical models to look for clusters. These clusters are what the scientists are talking about when they say that a region of the brain “lights up.” Now, though, those clusters are in question, because they may just be software errors.
What does this all mean? If this study is correct, then it means that the results from many fMRI studies from the last 20 years aren’t as certain as we thought they might be, and that not much can be done about it. “It is not feasible to redo 40,000 fMRI studies,” concludes the study, “and lamentable archiving and data-sharing practices mean most could not be reanalyzed either.”
The answer, then, is to make sure future results are cleaner. One way to do this is to keep using existing software, but run it thousands of times over on each data sample, to eliminate statistical errors—similar to measuring a piece of wood two or three times before you cut it, just to make sure you measured it right. And the other lesson is, of course, to remember that our view of the world is colored by the tools we use to look at it.
Have something to say about this article? You can email us and let us know. If it’s interesting and thoughtful, we may publish your response.