Taking Snow Measurements

Regressing to Prove a Point

“Believe me” is a popular phrase lately. It flows freely from people working to prove a point. It works for someone with something to sell. But it doesn’t work well in scientific journals. Consider this case of a pilot weight intervention study for an older lesbian population.

A Big Leap for Pilot Study

The SHE (Strong, Healthy, and Energized) program received funding as a pilot study in 39 participants. Lesbian and bisexual women over 60 joined 12 sessions focused on exercise, nutrition, stress management, and group discussions. With just 39 participants, the investigators reached some strong conclusions: a significant reduction in waist circumference (3.7%) and a recommendation to adopt this program in senior centers for lesbian and bisexual women broadly. Done.

One Little Problem: Regression to the Mean (RTM)

The only problem is that the effect observed in this study could be explained by something called regression to the mean (RTM). In a study like this one, which lacks a control group, people with extreme values at the start of a study will naturally tend to have less extreme values at the end of a study. It’s a common problem in obesity studies and we’ve discussed it here before.

It’s really not surprising that study of just 39 people could not prove efficacy for a new weight management program. That’s how it goes with pilot studies. The object of a pilot is to work out the bugs in a program, not to prove a point.

Intent on Proving a Point

But these investigators seem intent on proving a point. Other scientists, led by Tanya Halliday, pointed out the flaws in the original analysis in a letter to the journal’s editor last week. They explained that RTM, rather than effectiveness explain the study results. They went on to say:

This does not mean that the SHE program is not effective, only that it was not convincingly shown to be effective by ordinary scientific standards in this study.

Authors of the original study seem undeterred. They thanked the letter writers and moved on to explain their “multiple reasons” for believing in their program. Among those reasons were the inspired comments of focus group participants.

Belief is no substitute for scientific evidence. The SHE program many indeed be a fine program. But selling it based on weak evidence is a bad idea.

Click here for the original study and here for the letter to the editor, along with the original authors’ response.

Taking Snow Measurements, photograph © Washington State Department of Transportation / flickr

Subscribe by email to follow the accumulating evidence and observations that shape our view of health, obesity, and policy.


 

December 10, 2017