Seeing What We Want to See in Soda Policy

Objectivity is having a rough time these days. This is true whether the subject is politics, policy, or even a study in a medical journal. Very often, believing is seeing. Not the other way around. A new study on soda policy in JAMA Internal Medicine provides a case in point.

Improving Metabolic Profile Very Effortlessly

The IMPROVE study comes from the University of California at San Francisco, led by Elissa Epel. Robert Lustig (who famously declared “sugar is toxic”) was part of the research team. UCSF declared a ban on campus sales of sugar-sweetened beverages (SSBs) back in 2016. So researchers wanted to see if the ban would lead to better metabolic health for employees.

This was a pre-registered study with prespecified endpoints. The primary outcome measure for the study was HOMA-IR. It’s a sophisticated measure of insulin resistance. The secondary measures were waist-to-hip ratio and self-reported SSB consumption. In addition, the researchers listed five “other” outcome measures.

The study had no control group because the researchers didn’t have time to find one. So it’s a before and after study, which has some inherent weaknesses.

Primary and Secondary Outcomes

Did the ban lead to improvements in the primary outcome, insulin resistance? Well, that depends on how you look at the data. In the pre-post analysis, researchers found no significant difference. The p-value was 0.33 – not even close to the threshold of 0.05.

The investigators did, however, find a correlation between reporting less soda consumption and improvements in insulin resistance. So when people said they cut their soda consumption, their insulin resistance had improved. A correlation. Not a cause and effect relationship with the ban on SSBs.

On the secondary outcomes, the results are even murkier. Waist-to-hip ratio didn’t improve. But for reasons the researchers don’t explain, they skipped over that. Instead they focused on waist circumference. Maybe because they found an improvement there. After the ban, it went down by an average of about 2 cm. That’s less than an inch. But the p value was 0.01. So they counted it as a win, even though it wasn’t a prespecified outcome measure.

The other prespecified secondary outcome measure was self-reported SSB consumption two weeks after this soda policy took effect. For whatever reason, researchers went with 6-month and 12-month reports instead and they found a reduction. Honestly though, when something’s been banned from a workplace, should we be surprised that people report drinking less of it? As the investigators note, self-reports are a key limitation of this study.

Seeing What You Want to See

The headlines about this study are enthusiastic. “Removing sweet drinks from the workplace works,” says the Daily Republic. In a UCSF news release, senior author Laura Schmidt made a bold claim:

This is one potential workplace solution to the growing epidemic of obesity-related disease in America that is easy to pull off.

Maybe taking sugar-sweetened beverages out of the workplace is a good idea. As a matter of principle. Why not? But this study is not scientific proof that health benefits will ensue. In fact, it’s a study with many limitations. Just look at the list of them in the manuscript. Then consider that the primary endpoint – insulin resistance – didn’t improve in the before and after analysis.

People will see what they want to see in this study of soda policy. But an “easy solution” for obesity? Objectively, there’s simply no evidence for that conclusion. And if we continue to promote such wishful thinking, the goal of actually reducing the impact of obesity on health will slip further from our reach.

Click here for the study and here for more from the New York Times.

Antique Coke Machine, photograph © ajmexico / flickr

Subscribe by email to follow the accumulating evidence and observations that shape our view of health, obesity, and policy.

November 1, 2019