Mistakes Were Made (But Not by Me)

Mistakes happen. Every scientific study has weaknesses. Research on obesity, nutrition, and physical activity is hardly exempt. Sometimes errors and flaws are only discovered after publication. Usually the errors are handled gracefully. An erratum is published and sometimes a paper is withdrawn. But even smart scientists are susceptible to the human impulse for self-justification. The impulse is aptly described by Carol Tavis and Elliot Aronson in Mistakes Were Made:

Most people, when directly confronted by evidence that they are wrong, do not change their point of view or plan of action. They justify it even more tenaciously.

If you doubt this, consider two recent examples.

Izumi Aiso and colleagues published a 2014 paper in Lipids in Health and Disease with a title that conveyed their primary conclusion:

Compared with the intake of commercial vegetable juice, the intake of fresh fruit and komatsuna (Brassica rapa L. var. perviridis) juice mixture reduces serum cholesterol in middle-aged men: a randomized controlled pilot study

The only problem is that the conclusion was not supported by the data and its analysis. David Allison and colleagues published a detailed explanation of the statistical flaws in a commentary published by the original journal last week:

Although the article concludes that there are some significant benefits to their komatsuna juice mixture, these claims are not supported by the statistical analyses used. An incorrect procedure was used to compare the differences in two treatment groups over time, and a large number of outcomes were tested without correction; both issues are known to produce high rates of false positives, making the conclusions of the study unjustified. The study also fails to follow published journal standards regarding clinical trial registration and reporting.

Faced with these observations, the authors responded that their conclusion should be “moderated” to state that their juice mixture “may be effective.” The original claims remain published in the journal.

The second example comes from weaknesses observed in an uncontrolled study, published in Obesity, regarding fructose restriction by Robert Lustig and colleagues. They claimed to have shown dramatic cardiometabolic benefits in children with obesity simply by restricting fructose consumption – independent of any weight change.

About this study, Tauseef Khan and John Sievenpiper wrote to the editors of Obesity, saying:

We believe the observed effects on cardiometabolic risk factors are remarkably consistent with dramatic weight loss. This small uncontrolled study cannot overturn consistent results of systematic reviews and meta-analyses of controlled trials showing that fructose-containing sugars in isocaloric exchange with other carbohydrates do not lead to weight changes or clinically meaningful adverse effects on metabolic parameters with a signal for harm seen only when fructose-containing sugars provide excess calories.

Lustig responded, “We are unmoved by their arguments.”

Indeed, even scientists will sometimes twist themselves into pretzels to avoid facing weaknesses and errors in their own work.

Click here to read the study by Aiso et al, click here for the commentary by Allison et al and the response from the original authors. Click here for the study by Lustig et al, here for the letter from Khan and Sievenpiper, and here for the response from Lustig. For more on the book by Tarvis and Aronson, click here.

Looking Back Is Not Easy, photograph © Aftab Uzzaman / flickr

Subscribe by email to follow the accumulating evidence and observations that shape our view of health, obesity, and policy.

April 21, 2016