Pharma R&D Today
Ideas and Insight supporting all stages of Drug Discovery & Development
The Costs of Irreproducible Biomedical Research
Posted on November 28th, 2016 by Thibault Geoui in Pharma R&D
Issues with reproducibility plague basic biomedical research. In a June 2015 study published in PLoS Biology, the authors suggested that U.S. researchers spend $28 billion a year on research that is irreproducible.
This number, however, is just an estimate, and the true cost of money lost on irreproducible research could be much higher. More specifically, the authors estimate that 53 percent of all research is irreproducible and suggest that this number could be anywhere between 18 percent and 89 percent. In other terms, the cost of irreproducible research could be anywhere between $10 billion and $50 billion each year. On a related note, researchers at pharmaceutical companies have suggested that more than 75 percent of all conclusions found in peer-reviewed studies could not be replicated.
Fifty billion dollars wasted on irreproducible research — research that not only results in financial losses but also undermines the evidence basis of all biomedical research and inspires further research that’s fatally flawed — is a major concern of the U.S. government, research institutions and publishers everywhere. For instance in November 2015, the NIH encouraged journals to adopt guidelines that improve rates of reproducibility. Moreover, the Global Biological Standards Institute (GBSI) issued the Reproducibility2020 challenge that encourages the research community to find solutions that increase reproducibility of biomedical research by 2020.
Irreproducibility refers to errors or omissions that make the replication of experimental findings impossible. Factors that likely contribute to irreproducibility include poor reagent and reference materials, flaws in study design and laboratory protocols, and problems with data analysis and protocols. Of these problems, poor materials may be the biggest single factor undermining reproducibility, and researchers have focused their attention on this factor. In particular, cell line authentication — i.e., short tandem repeat (STR) profiling — has been a focus of various efforts and has received much press. However, it’s very likely that, with many studies, a combination of factors results in irreproducibility, further complicating this phenomenon and obfuscating potential solutions.
GBSI is a leader in identifying and developing solutions for the issue of irreproducibility in biomedical research. According to the GBSI website, the organization “is dedicated to enhancing the quality of biomedical research by advocating best practices and standards to accelerate the translation of research breakthroughs into life-saving therapies.”
The GBSI Reproducibility2020 action plan takes a three-pronged approach to solving the problem of irreproducibility:
- Reagent validation. Biological reagents should be validated and standardized in a fashion that changes the practices and expectations of researchers. For instance, cell line authentications should become the norm. Furthermore, the research community should establish antibody validation standards and create an open online registry that shares biospecimen procurement SOPs.
- Shared protocols and data. Researchers should share detailed protocols and data with other members of the research community. The development of novel technological tools, systems and dashboards will facilitate increased sharing.
- Improved training. Interactive training modules that are free and online should be developed and disseminated. This training should focus on experimental design and reagent validations for principal investigators, faculty and other interested stakeholders.
In addition to promoting these three efforts, GBSI will also examine other scientific factors that contribute to reproducibility and provide updates to the research community. Finally, GBSI will look at policy issues contributing to irreproducibility.
GBSI plans to measure the progress in reducing irreproducibility by collecting data using annual surveys and measuring other metrics. GBSI will spotlight researchers dedicated to change, and the organization will publish an annual report that details progress towards meeting its Reproducibility2020 priorities.
On a positive note, the research community has improved its understanding of irreproducibility and is now making efforts to tackle this pervasive issue. A mere decade ago, reproducibility was less recognized as a critical concern. In part, the increased attention to the issue is attributable to a landmark essay published in PLoS Medicine by John P.A. Ioannidis titled “Why Most Published Research Findings Are False.” Ioannidis suggested that publishers desire to print articles biased towards flashy and positive results likely further contributes to irreproducibility.
All opinions shared in this post are the author’s own.
R&D Solutions for Pharma & Life SciencesWe're happy to discuss your needs and show you how Elsevier's Solution can help.
- Helping Startups in the Pursuit of Drug Discovery Success
- Translational Medicine, Data Access and Pharmaceutical Innovation
- How Pharma Can Make the Most of Gene Tech CRISPR
- Extreme Makeover: Clinical Trials Edition
- As Drug Development Costs Skyrocket, AI Offers Some Hope