Spin, Pilots, and Sacred Cows of Obesity Care

Beware of pilot studies with claims of effectiveness. “A pilot study is not a hypothesis testing study,” says Andrew Leon in a cogent summary of what pilots can and can’t do. But it’s oh so tempting to jump on effect data from a pilot study. Especially when you believe in what you think it tells you.

Case in Point: Motivational Interviewing

Clinicians rely on motivational interviewing as an effective technique for counseling patients and leading them to make recommended behavior changes. You might call it a sacred cow for obesity care. The technique enjoys that status for good reasons.

By itself, though, it’s not a guarantee for health outcomes. It merely helps people follow recommendations. The effectiveness of the recommendations themselves is a critical factor.

Nonetheless, we note claims that “MI might be an effective adjunct to adolescent obesity treatment” in a recent paper by Melanie Bean and colleagues. They published this pilot study in Clinical Obesity. The title of the paper talks about the “impact” of MI on adolescent obesity treatment outcomes.

In a letter to the journal’s editor, Corby Vorland et al point out that this study actually does not demonstrate an effect on obesity. “Conclusions about effects on obesity based only on behaviour change (whether self-reported or not) are unwarranted,” they write.

To their credit, the authors responded well:

We should have explicitly stated in the abstract, as we did within the body of the paper, that there were no between group differences in BMI z-score.

Even better, they write that they will modify their publication abstract accordingly.

Pilots Are About Feasibility, Not Efficacy

Stepping back from the numbers here, it’s important to go back to a more basic point. Pilot studies are all about feasibility. Are the methods appropriate for testing this intervention in a larger study? That’s the question a pilot study answers.

So if you see efficacy words like “impact” in the title of a report on a pilot study, the authors are likely overstating their case. Claims of efficacy from pilots come with a huge grain of salt.

Click here for the study, here for the Vorland letter, and here for the response.

Pilot, photograph © Ignacio Ferre Pérez / flickr

Subscribe by email to follow the accumulating evidence and observations that shape our view of health, obesity, and policy.

August 15, 2019