Finding a Confession of Faith in a Dataset

Finding a Confession of Faith in a Dataset

Knowing what you believe can be extremely important. But for science, that’s not enough. There’s a big difference, for example, between believing that intermittent fasting can help patients with diabetes and measuring the clinical outcomes that it produces. Likewise, believing that an Active Classroom intervention can boost activity levels at school is not the same as knowing that it does. Faith in an idea can be a good thing. But to know if that idea is sound, you must test it.

A Study of Active Classrooms

The authors of a recent study of an Active Classrooms intervention write that the primary strength of their study was the idea they tested. That’s because behavior change theory is the basis for developing their intervention. It engages the beliefs, attitudes, and and perceptions of classroom teachers who must implement the program.

Indeed, that sounds like a good idea. But does it work?

The answer to that question is a bit murky. The results of the study were, at best, mixed. The authors claim to have seen an impact on one of their primary outcomes – physical activity in class. But they saw absolutely no effect on total physical activity throughout the day (p=0.52). They dismiss that finding with an explanation that makes no sense to us:

While differences between the groups were not statistically significant, results indicate a large intervention effect size on students’ whole school day MVPA levels. The large effect size in absence of a statistically significant result is a relatively common occurrence and may be due to small sample size or large variability. Nonetheless, the effect size noted in the current study gives an indication of the magnitude and practical significance of the treatment effect.

We really don’t understand how outcome that is “not statistically significant” different can indicate an effect size with “practical significance.” Strong belief in a concept is not a substitute for proof that it works.

Turning Anecdotes into Evidence?

In another example, we wrote last week about the promise of intermittent fasting. But we noted that good proof for an effect on long-term outcomes is lacking. In particular, we took issue with an assertion by Jason Fung that a case report of three patients “changes everything about how we should treat type 2 diabetes.”

Fung was not happy. So he summoned his fans on Twitter:

Let’s ask Twitter. Has anybody out there been able to use fasting to reduce medications for type 2 diabetes?

The sum of anecdotes is not evidence. Case reports can be useful for generating ideas. But ideas have to be tested. And that testing needs good controls. When you lack controlled observations, issues like regression to the mean pop up. Though it’s a common problem, many folks ignore it.

Faith in your beliefs can be a good thing. But when it leads people to disregard facts and evidence, it puts them at risk.

Click here for the Active Classrooms study. For an investigation of why people hang on to false beliefs, click here.

Subscribe by email to follow the accumulating evidence and observations that shape our view of health, obesity, and policy.

November 1, 2018