12/13/2021
As scientists work to advance medicine, finding new cures and treatments relies on immense amounts of research. Unfortunately, research only goes so far when the results cannot be replicated. Over the past eight years, the Reproducibility Project: Cancer Biology has tried to replicate 193 experiments from 53 cancer research papers, but only succeeded on a quarter of them. And without reproducible results, it is difficult to move forward with these findings.
There were many reasons for lack of replication. Sometimes the team could not gather enough information from the research papers about the exact methods used, or could not acquire the necessary materials. Sometimes authors of these papers were unable to be reached, or outright refused to help. Others had simply lost necessary information on misplaced hard drives or because of intellectual property restrictions. No matter the case, these studies had irreproducible results, and could hinder development going forward.
Of the studies that were reproducible, effect sizes were, on average, 85% lower than that of the reported effect size in the original study. And on five criteria that the team used to measure the success of the replication, only 46% passed. Again, this does not bode well for further innovations.
This certainly does not only happen in cancer research. In a study from Nature, out of 1,576 researchers, more than 70% have tried and failed to reproduce another scientist’s experiments, and over half failed to reproduce their own. And even though 52% agree there is a reproducibility crisis, only 31% think that failing to reproduce published results means that the result is wrong. Even if replication is not necessary for results to be valid and true, any further work based on these results requires confirmation of valid results beyond the initial study.
While it may be easy to say that scientists should ensure some level of reproducibility in their studies, the blame may be equally on the institutions and culture around these studies. For researchers, publication leads to advancements in funding, new jobs, or keeping their current job. Publishing innovative research fits into this reward loop, even if it cannot be replicated. Conducting your research twice as long for reproducibility or sharing your findings so another researcher can reproduce your work does not.
Rather than encouraging privatization of research such that scientists are disincentivized from sharing their findings before publication, governments and organizations should encourage researchers to work together to ensure validity. Another avenue is to remove the intense level of competition for gaining funding in favor. Or on an even more basic level, stop funding research that cannot be replicated.
The apparent replication crisis may be the warning that research needs to change. Whether this change is institutional or individual, any action to demystify the results of respected publications is a step in the right direction.