Impaired Analyses and Overreaching Claims
An appealing narrative is seductive. Recently, we tripped over a case study in this basic fact when a new study in Nature Metabolism stirred up considerable attention from health reporters with claims about “severely impaired” brain responses to nutrients in humans with obesity. But in retrospect, there’s an plausible argument that the analyses were impaired and inadequate for making such overreaching claims.
Impairment?
Impairment is, by definition, a negative change from a normal state of function. Documenting impairment requires a comparison to the normal state and it turns out that the investigators have not done a statistically valid comparison. Kevin Hall (a scientist not involved with this study) explains:
“Does this elegant study demonstrate what the authors claim? I can’t find any direct statistical comparisons of the brain responses in people with and without obesity, so I don’t see how these conclusions are justified.”
Further, he reports that:
“The authors confirmed my understanding of their analyses saying ‘we do not claim anywhere in the manuscript that we directly compared the lean to the obese group.’
“Readers of the paper or its press coverage might be surprised given the strong claims about obesity impairments!”
So, no. It’s not clear that this study documents an impairment of brain response to nutrients. Because we can find no valid comparison to what is normal.
An Illusory Control Group
At first glance, this study gives the appearance of being well-controlled, though it is small. They have 30 subjects with obesity and 30 subjects without. They find a response to nutrients in the lean subjects. But no response in the subjects with obesity. The measurement of the response is based upon within-group comparisons between fat or sugar infusions to a non-nutritive control infusion.
The investigators originally (in their trial registration) planned to compare subjects with obesity to lean subjects. But according to the study design disclosure attached to the final publication, they later realized they did not have a sufficient sample size to make such a comparison valid.
So we are left with an implicit comparison in the claim that obesity impairs brain responses to nutrients in persons with obesity. In this way, investigators allow readers to assume the research provides a valid comparison to lean subjects. But unfortunately, it does not.
When the control group disappeared from their statistical analysis, the claim of impairment should have disappeared, too. Because a valid claim of impairment requires comparison to a normal state.
We regret being misled by that study’s false claim when we originally wrote about it. The neuroscience of obesity is important. Researchers should report their findings with care and fidelity to the whole truth of what they find and what they do not.
Click here for the study in Nature Metabolism.
Eight Windows Stretch-Wrapped in Plastic, photograph by W.carter, licensed under CC BY-SA 4.0
Subscribe by email to follow the accumulating evidence and observations that shape our view of health, obesity, and policy.
June 19, 2023
June 19, 2023 at 3:37 pm, Angie Golden said:
Ted, thank you for this update. As always you make transparency a REAL thing.