Reproducibility of Science: Look Twice Before Crossing

Some call it a crisis of reproducibility. More than a decade ago, John Ioannidis famously told the world that most published research findings are false. His analysis quickly became the most widely read paper ever published by PLOS. You’ll find a more generous view in a new, special issue of PNAS.

Attending to the rigor, reproducibility, and transparency of science is a never-ending process, says Professor David Allison. Together with Richard Shiffrin, Victoria Stodden, and Stephen Fienberg, Allison organized the Sackler Colloquium that became the basis for this special issue.

Prevent, Detect, Admit, Correct

Allison sees many opportunities to increase the veracity and usefulness of science. He tells us:

Recently we have noticed an apparently high frequency of substantial errors in literature, especially around statistical analysis and interpretation. Under the banner of prevent, detect, admit, correct, we are working in both our own research and to help others to prevent errors. And when errors do occur, to detect them, acknowledge them, and publicly correct them. Anecdotally, we are having substantial success in preventing errors before publication via statistical checks and via statistical education.

Detecting many errors is not difficult. But our recent experience tells us that many authors and journal editors seem unwilling to acknowledge and rectify errors – even when those errors are unequivocally noted. Creating incentive systems to promote the unvarnished acknowledgment and correction of errors is an important challenge for the field going forward.

Crisis or Ongoing Responsibility?

For every serious scientist, this entire collection is worth reading carefully. It covers the range of issues facing scholars concerned with scientific integrity and rigor – completely and concisely. But for an even broader audience, two papers in the collection are especially noteworthy.

Daniele Fanelli presents a crisp analysis to refute the idea that science faces a crisis of irreproducible results. He concludes that the crisis narrative is both mistaken and unhelpful. More accurately, he says, science is on the cusp of a revolution in transparency and reproducibility.

Kathleen Hall Jamieson examines the same question through the lens of media narratives. She concludes that the portrayal of science in crisis is inherently misleading. News can serve its function to ensure accountability better, she says, by responsibly reporting on both breaches of integrity and extensive work to prevent them.

This tension between stories of crisis and the quest for integrity and rigor will likely be with us forever. Is the glass half empty or half full? People will see the issues as they wish. But what matters is our best efforts to address them. We must, as Jamieson suggests, protect public trust in the most reliable form of knowledge generation that humans have devised – science.

Click here for an introduction to this collection and here for the table of contents. For a more cynical view, read this commentary by George Lundberg.

Deer Crossing, photograph by Julie Falk, licensed under CC BY-NC 2.0

Subscribe by email to follow the accumulating evidence and observations that shape our view of health, obesity, and policy.

March 19, 2018