a guide to debunking bad science

Tired of listening to the news and hearing about the latest health scare or supplements and wondering if you should be worried? Here's a few questions to ask to help you wade through all the science. short version: if it's not an actual experiment done on humans and you dont have time to wade through all the science then just ignore the study. long version: 1. Did the news article or story cite its source from a peer reviewed journal? If it's an article on the internet it is very easy to just click on the links that are provided, scroll down and copy paste provided links or citations, or google scholaring the author's name. If not a peer reviewwed journal ignore. If yes go to next question a. Is the source an actual study or is it simply preliminary result presented at a conference? If the latter please be aware that preliminary results dont always pan out. You want a study that has been peer reviewed. b. How old is the source? Simply being old does not necessarily mean the study is a bad one. However older studies have a few disadvantages. Certain techniques might have turned out to be not very effective or very error prone. Studies may have since shown that some of the assumptions or some of the background was wrong. What you really want to see in the article is a study published in a peer review journal, not more than a decade or two old. If its "news" the study should actually be fairly recent, within the past few years. The more famous the journal (nature, science) the more impact and newsworthy the article might be (not necessarily, but more likely). That is not to say the study is more credible if in a famous journal. 2. Was there an actual experiment or did the study simply observe something? If it simply observed something (eg population studies) then keep in mind that the study cannot prove anything. All it can do is make the link between A and B, but it cannot say A caused B. If there was an actual experiment go to the next question 3. Was the study done in vivo (in a whole living organism) or in vitro (outside a living organism eg in a test tube)? If it was in vitro keep in mind that the experiment might not apply in real life as organisms are complex beings with many factors affecting the subject in hand. In other words while the experiment might help elucidate the subject keep in mind that it only isolated and studied one part of the mechanism. If in vivo on to next question 4. Was the study done in humans or non humans (eg mice, rats, yeast, bacteria, flies)? If in non humans keep in mind that non humans are not humans and dont have the same metabolic pathways as humans and the experiment might not be applicable. Another thing to keep in mind is dosing. Did the non humans receive an amount that is typically consumed by humans or did the scientist subject them to ridiculous amounts of the subject matter in an attempt to gain publicity and/or grants? If in humans on to the next question 5. Is the study applicable to you? Is the dosing realistic? Many times news articles will cite health problems for people who are way way overexposed to the subject in an attempt to discredit the subject. Figure out how much of the subject material they are using to get these results and ask your self can I reasonably get this amount? Is the timing realistic? For example one study tried to show that post workout nutrition did not improve anything, but if you looked at the study they spread nutrition dosing out over 8 hours after the workout, which is not what most ppl think of as post workout nutrition. Are the subjects in a similar health condition to you? Many times scientists will be studying a specific part of the population and people will try and use those studies out of context. If you are not a diabetic then the study on diabetic patients might not be applicable to you. Have the experiments been replicated and produced similar results? If it was a one off study that disagrees with the vast majority of studies than it is most likely that there is some other variable in that study that influenced the outcomes. Questions 1 takes you 5 seconds to scroll down to the bottom of the page to see if there are citations. Questions 2 to 4 takes you 30 seconds to look for in the abstract (Was it a population study? What were the subjects of the study (in vitro? rats? humans)? Questions 5 and on will take some reading to do through the methodology, but a quick scan of the abstract usually provides enough information. If you got to this point and all the answers were good then you should start to wonder whether or not the premise is true, but until then you dont have much. here's a fun seminar by tom noughton on smart science
IMPORTANT NOTICE: No media files are hosted on these forums. By clicking the link below you agree to view content from an external website. We can not be held responsible for the suitability or legality of this material. If the video does not play, wait a minute or try again later. I AGREE

TIP: to embed Youtube clips, put only the encoded part of the Youtube URL, e.g. eBGIQ7ZuuiU between the tags. here's a good website on debunking health scares. here's a good book by the same author

Here are some quick contributions. I'm sure I will think of more, soon. WEB: http://theness.com/neurologicablog/ http://www.sciencebasedmedicine.org/ http://www.quackwatch.com/ http://www.badscience.net/ BOOKS: Denialism Bad Science

nice. will spending some time reading them

Originally Posted by indesertum

Click to expand...

nice. will spending some time reading them

Denialism is a great book and can be had for cheap on Amazon.

Here are some podcast recommendations for science and critical thinking that often cover quackery:

http://www.theskepticsguide.org/sgu....terPodcastId=1 - My favorite.

http://skeptoid.com/

http://www.skepticality.com/

http://moremark.squarespace.com/quackcast-home/

This is far too cut and dry. In vitro experimentation has provided countless scientific observations that have been crucial to... everything (no source for that claim, but relevant nonetheless). Often times, it is not in setting stringent guidelines, but rather just looking at the basic abstract that a scientific experiment (typically presented in a journal). IF an experiment presents too many variables to control, if the data collection is poor, or other things, these may be cause for concern. These questions that you pose are admittedly horribly ignorant.

Constant development, trial and error, and the experiments you would tend to ignore provide both the ideas behind and the basic experimental techniques for the in vitro super experiments (that make up a minority of actual contemporary scientific work) that you propose are the only legitimate experiments. I would advise all reading to take the aforementioned points with a serious grain of salt.

its meant to be cut and dry. most ppl dont have time to wade through one study much less all studies.

the guide isnt meant for you to understand the science fully, but it works in weeding out all the bullshit from the ton of health scare stories you hear every day.

insect studies, rat studies, population studies are important in pointing out the questions that need to be researched. in vitro studies are important and indeed crucial in understanding the underlying mechanism, but to prove that it impacts humans in a meaningful way you need a clinical trial on humans.

simply citing a few rat studies or in vitro studies is not adequate in proving something like "cell phones kill brain cells", which is unfortunately what the vast majority of media does

Also, research supporting the benefits of a given supplement should not only be from human research, it should be published in peer-reviewed journals. With supplements, they almost never are.

It may also be of importance to be able to separate politics from science as well. In America, it's usually the policy that is driving the science, and not the other way around.

Originally Posted by mm84321

Click to expand...

In America, it's usually the policy that is driving the science, and not the other way around.

[citation needed]

I've seen a lot of evidence of ****** science reporting driven by politics, but the bleed through to the literature is very minimal at most. I really don't think that science is driven by politics, beyond issues like more money going to certain fields vs others. I don't think it influences the veracity of the results in the end.

You're taking your chances ready any secondhand discussion, and people should recognize that. Important point to make.

Originally Posted by Gibonius

Click to expand...

[citation needed] I've seen a lot of evidence of ****** science reporting driven by politics, but the bleed through to the literature is very minimal at most. I really don't think that science is driven by politics, beyond issues like more money going to certain fields vs others. I don't think it influences the veracity of the results in the end.
Maybe not the veracity of the results themselves, but the interpretation, or rather, misinterpretation, of the data certainly can be motivated and biased by policy and corporations to best suit their needs. Senator George McGovern and the formation of the Food Guide Pyramid in 1976 comes to mind. The USDA's recommendation that a "healthy diet" for Americans consists of 7-10 servings of grains per day was based on what scientific evidence?

Originally Posted by mm84321

Click to expand...

Maybe not the veracity of the results themselves, but the interpretation, or rather, misinterpretation, of the data certainly can be motivated and biased by policy and corporations to best suit their needs. Senator George McGovern and the formation of the Food Guide Pyramid in 1976 comes to mind. The USDA's recommendation that a "healthy diet" for Americans consists of 7-10 servings of grains per day was based on what scientific evidence?
That's a very different thing from "policy drives science." The government has never been the arbiter of scientific validity. Saying "policy drives science" indicates collusion on the part of scientists to support political policy decisions, which is certainly not the case in that scenario.

Originally Posted by Gibonius

Click to expand...

That's a very different thing from "policy drives science." The government has never been the arbiter of scientific validity. Saying "policy drives science" indicates collusion on the part of scientists to support political policy decisions, which is certainly not the case in that scenario.

I suppose I shouldn't have said it was policy driving science, but more that it is science that doesn't drive policy. Although, the types of funding given for certain studies is largely driven by politics and financial interest groups, so, in a way, science itself is being made possible by policy.

Originally Posted by Gibonius

Click to expand...

[citation needed]

I've seen a lot of evidence of ****** science reporting driven by politics, but the bleed through to the literature is very minimal at most. I really don't think that science is driven by politics, beyond issues like more money going to certain fields vs others. I don't think it influences the veracity of the results in the end.

You're taking your chances ready any secondhand discussion, and people should recognize that. Important point to make.

I agree that this is the real problem. The reporting on scientific issues in this country is near pathetic.

People just need to learn to examine studies with a critical eye. If it's big enough news that you might change your life based on it, it's worth examining, like the bullshit sunscreen and melanoma thing.

Originally Posted by indesertum

Click to expand...

here's a fun seminar by tom noughton on smart science
IMPORTANT NOTICE: No media files are hosted on these forums. By clicking the link below you agree to view content from an external website. We can not be held responsible for the suitability or legality of this material. If the video does not play, wait a minute or try again later. I AGREE

TIP: to embed Youtube clips, put only the encoded part of the Youtube URL, e.g. eBGIQ7ZuuiU between the tags.
I'm watching this now. Pretty good. I also saw Noughton's Fat Head movie. An article cited by Noughton in his video: Lies, Damned Lies, and Medical Science The interesting parts: "(Ioannidis) and his team have shown, again and again, and in many different ways, that much of what biomedical researchers conclude in published studies—conclusions that doctors keep in mind when they prescribe antibiotics or blood-pressure medication, or when they advise us to consume more fiber or less meat, or when they recommend surgery for heart disease or back pain—is misleading, exaggerated, and often flat-out wrong. He charges that as much as 90 percent of the published medical information that doctors rely on is flawed." "Simply put, if you’re attracted to ideas that have a good chance of being wrong, and if you’re motivated to prove them right, and if you have a little wiggle room in how you assemble the evidence, you’ll probably succeed in proving wrong theories right. ((Ioannidis's) model predicted, in different fields of medical research, rates of wrongness roughly corresponding to the observed rates at which findings were later convincingly refuted: 80 percent of non-randomized studies (by far the most common type) turn out to be wrong, as do 25 percent of supposedly gold-standard randomized trials, and as much as 10 percent of the platinum-standard large randomized trials."