The Painful Walk Away from a Flawed Analysis

The Painful Walk Away from a Flawed Analysis

Five months ago, we wrote about inflated claims of effectiveness from a pilot study of obesity prevention by Scherr et al. An independent group of researchers had written to the journal with concerns about the flawed analysis of the study. The flaws effectively canceled out the claims of effectiveness for the program. But Scherr et al simply brushed off the concerns.

The Story Continues

However, the concerns didn’t go away. Late last week, the journal published yet another letter outlining concerns about this flawed analysis. Sean Lucan wrote that valid conclusions can only come from valid analyses:

So, in my opinion, the result is that their findings of reductions in BMI percentile may have been inflated or, more likely, just plain false.

The bottom line is that the currently published work cannot stand uncorrected as part of the extant literature. My opinion is that the published article is invalid and must be corrected or retracted.

A Bit of a Correction

This time, the authors’ response was a bit more substantive:

We acknowledge the points raised in this letter. Based on the comments, additional analyses were conducted on the full data set. These results, along with some amended statements, can be found in a corrigendum published in this journal.

But they’re still hanging onto that claim of effectiveness for their intervention. They just watered it down a little. In the original abstract the authors made an unqualified claim of “a significant decrease in BMI percentiles.”

A Qualified Claim of Effectiveness

Now, they’ve modified the conclusion in the abstract. They still say their program produced lower BMI percentiles. But only in the one district that “fully implemented” the program.

At the end of the day, we simply cannot figure out how Scherr et al have supported this conclusion. CONSORT guidelines tell us:

Trials with one cluster per arm should be avoided as they cannot give a valid analysis, as the intervention effect is completely confounded with the cluster effect.

But that’s precisely what Scherr had left in the analysis of the one district that “fully implemented” her program – an intervention arm with only one cluster.

No doubt, this program “has promise,” as Scherr now says in her revised abstract. So the authors would do well to simply drop the unsubstantiated claim of effectiveness. Credibility would rise.

Click here for the original study and here for the corrigendum. For the letters to the editor and responses, click here, here, here, and here. The order is chronological.

Subscribe by email to follow the accumulating evidence and observations that shape our view of health, obesity, and policy.

August 10, 2018