Corruption of Science: The Reproducibility Crisis

One of the more obvious signs that modern science is ailing is the reproducibility crisis – the vast number of peer-reviewed scientific studies that can’t be replicated in subsequent investigations and whose findings turn out to be false. In the field of cancer biology, for example, researchers discovered that an alarming 89% of published results couldn’t be reproduced. Even in the so-called soft science of psychology, the rate of irreproducibility hovers around 60%. And to make matters worse, falsification and outright fabrication of scientific data is on the rise.

The reproducibility crisis is drawing a lot of attention from scientists and nonscientists alike. In 2018, the U.S. NAS (the National Association of Scholars in this case, not the Academy of Sciences), an academic watchdog organization that normally focuses on the liberal arts and education policy, published a particularly comprehensive examination of the problem. Although the emphasis in the NAS report is on the misuse of statistical methods in scientific research, the report discusses possible causes of irreproducibility and presents a laundry list of recommendations for addressing the crisis.

The crisis is especially acute in the biomedical sciences. Over 10 years ago, Greek medical researcher John Ioannidis argued that the majority of published research findings in medicine were wrong. This included epidemiological studies in areas such as dietary fat, vaccination and GMO foods as well as clinical trials and cutting-edge research in molecular biology. 

In 2011, a team at Bayer HealthCare in Germany reported that only about 25% of published preclinical studies on potential new drugs could be validated. Some of the unreproducible papers had catalyzed entirely new fields of research, generating hundreds of secondary publications. More worryingly, other papers had led to clinical trials that were unlikely to be of any benefit to the participants.

Author Richard Harris describes another disturbing example, of research on breast cancer that was conducted on misidentified skin cancer cells. The sloppiness resulted in thousands of papers being published in prominent medical journals on the wrong cancer. Harris blames the sorry condition of current research on scientists taking shortcuts around the once venerated scientific method.

Cutting corners to pursue short-term success is but one consequence of the pressures experienced by today’s scientists. These pressures include the constant need to win research grants as well as to publish research results in high-impact journals. The more spectacular that a paper submitted for publication is, the more likely it is to be accepted, but often at the cost of research quality. It has become more important to be the first to publish or to present sensational findings than to be correct.      

Another consequence of the bind in which scientists find themselves is the ever increasing degree of misunderstanding and misuse of statistics, as detailed in the NAS report. Among other abuses, the report cites spurious correlations in data that researchers claim to be “statistically significant”; the improper use of statistics due to poor understanding of statistical methodology; and the conscious or unconscious biasing of data to fit preconceived ideas.

Ioannidis links irreproducibility to the habit of assigning too much importance to the statistical p-value. The smaller the p-value, the more likely it is that the experimental data can’t be explained by existing theory and that a new hypothesis is needed. Although p-values below 0.05 are commonly regarded as statistically significant, using this condition as a criterion for publication means that one time in twenty, the experimental data could be the result of chance alone. The NAS report recommends defining statistical significance as a p-value less than 0.01 rather than 0.05 – a much more demanding standard.

The report further recommends integration of basic statistics into curricula at high-school and college levels, and rigorous educational programs in those disciplines that rely heavily on statistics. Beyond statistics, other suggested reforms include having researchers make their data available for public inspection, which doesn’t often occur at present, and encouraging government agencies to fund projects designed purely to replicate earlier research, which again is rare today. The NAS believes that measures like these will help to improve reproducibility in scientific studies as well as keeping advocacy and the politicization of science at bay.

Next week: Corruption of Science: Scientific Fraud