Vegetable Fields

Healthy Eating: Statistical Significance That’s Insignificant

Healthy eating patterns improve health. Right? So when a new study in Clinical Pediatrics tells us that kids with obesity who follow any of three healthy eating patterns will improve their cardiovascular health, should we believe it? Unfortunately not. Because in fact, the statistical significance of their findings about healthy eating reflects an error. Not genuine statistical significance.

In an interview with Healio, lead author Michael Macknin described this error, though without labeling it as an error:

“We initially undertook the study to try to compare the effectiveness of the three healthy eating patterns highlighted in the 2015-2020 Dietary Guidelines for Americans. For unclear reasons, our patient population, selected using identical criteria to our pilot study, had lab values closer to normal and worse compliance than our pilot study patients. Therefore, our study was not sufficiently powered to detect between group differences, so we reported changes within the groups.”

Regression to the Mean

What Macklin and his colleagues have done is to document something called regression to the mean. When a group of subjects in the study start out with abnormal values for something – in this case, markers of heart health – those values will naturally drift closer to normal during a study. It doesn’t really matter if you do something to make it better. High numbers come down. Low numbers come up. Not always a lot, but enough to fool researchers who don’t have a control group and don’t pay attention to this source of errors.

Moving the Goalposts and Proving a Point

To these researchers and to the casual reader, the findings make sense. These are all healthy eating patterns. Why shouldn’t they lead to better heart health?

But there are two problems with this line of thinking. First of all, the researchers moved the goalposts for their study. The original intent was to see if there was a difference in the effects of these three eating patterns. They found none. So they moved to making comparisons to baseline within each group to conclude that all three eating patterns are effective for improving heart health.

Unfortunately, they did not have a control group and thus regression to the mean might account for the changes they observed. In addition, as the authors note, the “study’s attention alone” might have led to the improvements seen.

Confirming Biases

The real problem here is that this study confirms biases rather than discovering the truth of how to improve the health of children with obesity. The bias going in is that children with obesity have bad diets. The assumption is that giving them better diets will solve the problem and make them healthier. But in fact, they had no evidence that the diets of these children were any better or worse than the average diets of children. In fact, their healthy eating index scores were similar to the scores USDA reports for all children.

So it’s likely that the diets of these children were like the diets of any others – some were healthier than others. Maybe these three diets helped some of them and had no effect for others.

Bottom line, this study tells us nothing. It serves only as an illustration of regression to the mean and confirmation bias.

Click here for the study and here for the report from Helio on it. For more on common errors in childhood obesity studies, click here.

Vegetable Fields, painting by August Macke / WikiArt

Subscribe by email to follow the accumulating evidence and observations that shape our view of health, obesity, and policy.


 

October 2, 2021

2 Responses to “Healthy Eating: Statistical Significance That’s Insignificant”

  1. October 02, 2021 at 7:34 am, Al Lewis said:

    Welcome to my world. The entire employee diabetees management industry is built on a combination of regression to the mean and participation bias and “moving the goal posts”

    https://theysaidwhat.net/2019/08/26/are-livongos-outcomes-real/

    • October 02, 2021 at 7:39 am, Ted said:

      Sadly true. Recent history has taught me that people can rationalize anything. Literally anything.