Is This Study Legit? Five Questions to Ask
Who doesn’t want to know if drinking that second or third cup of coffee a day will improve your memory, or if sleeping too much increases your risk of a heart attack?
We’re invested in staying healthy and many of us are interested in reading about new research findings to help us make sense of our lifestyle choices.
But not all research is equal, and not every research finding should be interpreted in the same way. Nor do all media headlines reflect what was actually studied or found.
So how can you tell? Keep these five questions in mind when you’re reading media stories about new studies.
1. Has the Research Been Peer Reviewed?
Peer review is a process by which experts in the discipline check a study to assess the its scientific validity.
This process involves the researcher writing up their study methods and results, and sending this to a journal. The manuscript then usually goes two to three experts for peer review.
If there are major flaws in a study, a journal will either reject it for publication, or require the researchers to address these flaws.
Although the peer-review process isn’t perfect, it shows a study has received scrutiny.
Read any reported findings that lack a peer review with a degree of reservation.
2. Was the Study Conducted in Humans?
Findings from studies in animals such as mice or on cells in a lab (in vitro studies) represent the earliest stage of the scientific discovery process.
Regardless of how intriguing they may be, no confident claims about human health should ever be made based on these types of study alone. There is no guarantee that findings from animal or cell studies will ever be replicated in humans.
3. Are Findings Likely to Represent a Causal Relationship?
For a study to have relevance to our day-to-day health, the findings need to reflect a causal relationship rather than just a correlation.
If a study showed that coffee drinking has an association with heart disease, for example, we want to know if this was because coffee actually caused heart disease. Or did these two things just happen to occur together?
In a number of studies that found this association, researchers subsequently found that coffee drinkers were more likely to be smokers and therefore, these results were more likely to reflect a true causal relationship between smoking and heart disease.
In observational studies, researchers observe differences in groups of people. It can sometimes be difficult to disentangle the relationship between variables.
The highest level of evidence regarding causality comes from double-blind placebo controlled randomised controlled trials (RCTs). This experimental type of study is where people separate into groups to randomly receive either an intervention or placebo (sham treatment). It is the best way we can determine if a something causes disease. However it, too, is not perfect.
Although other types of studies in humans play an important role in our understanding of health and disease, they may only highlight associations that are not indicative of causal relationships.
4. What Is the Size of the Effect?
It’s not enough to know that an exposure (such a third cup of coffee or more than nine hours of sleep a night) causes an outcome. It’s also important to clearly understand the strength of this relationship. In other words, how much is your risk of disease going to increase with exposure?
If your risk of disease reportedly increases by 50% (which is a relative risk), this sounds quite frightening. However, if the original risk of disease is low, then a 50% increase in your risk may not represent a big actual increased risk of disease. A 50% increased risk of disease could mean going from a 0.1% risk of disease to your risk being 0.15%. That doesn’t sound quite so dramatic.
5. Is the Finding Corroborated by Other Studies?
A single study on its own, even if it’s a well-conducted randomised controlled trial, can never be considered definitive proof of a causal relationship between an exposure and disease.
As humans are complex and there are so many variables in any study, we can’t be confident we understand what is actually going on until others replicate the findings in many different groups of people, using many different approaches.
Until we have a significant body of evidence that is in agreement, we have to be very careful about our interpretation of the findings from any one study.
What if These Questions Aren’t Answered?
If the media report you’re reading doesn’t answer these questions, consider changing news sites or looking at the original paper. Ideally this would be linked in the news article you’re reading, or you can search PubMed for the article using a few keywords.
The journal article’s abstract should tell you the type of study, whether humans were the subjects, and the size of the effect. If a paywall doesn’t block you, you may be able to view the full journal article. That should answer all of the questions you have about the study.
This article, by Hassan Vally, was originally published here on The Conversation under a Creative Commons license. Vally is an Associate Professor at La Trobe University in Melbourne, Australia.
The Reader, painting by Henri Matisse / WikiArt
Subscribe by email to follow the accumulating evidence and observations that shape our view of health, obesity, and policy.
October 15, 2019
