Sorting Out “What the Science Says” Is Not So Easy

There’s a new mantra making the rounds, but it’s really not so new. Let’s follow what the science says. That’s well and good, except that the science is seldom as definitive as we would like. In fact, when you dig into the details of any given study, you may find surprises – or more questions than answers. When you add in the phenomenon of motivated reasoning, then politics take over, and the tussle begins.

Right now, though, we’re going to focus on the primary challenge of understanding what a piece of research can and can’t tell us. A key tool for this is post-publication peer review.

Case in Point: Electroacupuncture for Simple Obesity

The volume of peer reviewed scientific literature has grown explosively. So tidy peer reviews before publication are often not adequate to spot and address questions that naturally arise. Let’s consider the case of a systematic review and meta-analysis on the effectiveness of electroacupuncture for simple obesity. (We’ll set aside the oxymoron of simple obesity.)

This treatment falls a bit out of the mainstream, so a systematic review and meta-analysis is welcome for the insights it might bring. The authors examined 13 studies with 937 patients. They concluded:

“For treating simple obesity, electroacupuncture is superior to other interventions such as acupuncture, acupoint catgut embedding therapy, and simple lifestyle modification for improvement in body fat rate, waist circumference, and waist-hip ratio, although not hip circumference.”

Sounds good, but the devil is in the details. Meta-analysis is a sophisticated tool that requires considerable care to produce reliable results. On PubPeer, a group of researchers led by Xiwei Chen from the IU Bloomington School of Public Health took a closer look. What they found was a need for corrections:

“Although many of the final conclusions in Gao et al remain the same in our re-analysis, the effectiveness of electroacupuncture is overstated and the analyses should be corrected. The original data from the papers and abstracts used in the meta-analyses should also be made publicly available.”

Peer Review: Sometimes Uncomfortable, Always Essential

Scientific insights are seldom cast in bronze. Especially now, with the volume of scientific literature exploding, peer review is essential and it does not stop with publication. When peer review shines a light on an error or an overstatement, it can be uncomfortable. But it should be welcome.

Because the whole point of scientific inquiry is to find reliable insights. Not to prove a point.

Click here for the paper on electroacupuncture and here for the analysis on PubPeer. For more on the challenge of knowing “what the science says,” click here and here.

Science, Labor, and Art, mural by Jose Clemente Orozco / WikiArt

Subscribe by email to follow the accumulating evidence and observations that shape our view of health, obesity, and policy.

February 21, 2021