Posted on 08/30/2005 10:29:44 AM PDT by LibWhacker
Most published scientific research papers are wrong, according to a new analysis. Assuming that the new paper is itself correct, problems with experimental and statistical methods mean that there is less than a 50% chance that the results of any randomly chosen scientific paper are true.
John Ioannidis, an epidemiologist at the University of Ioannina School of Medicine in Greece, says that small sample sizes, poor study design, researcher bias, and selective reporting and other problems combine to make most research findings false. But even large, well-designed studies are not always right, meaning that scientists and the public have to be wary of reported findings.
"We should accept that most research findings will be refuted. Some will be replicated and validated. The replication process is more important than the first discovery," Ioannidis says.
In the paper, Ioannidis does not show that any particular findings are false. Instead, he shows statistically how the many obstacles to getting research findings right combine to make most published research wrong.
Massaged conclusions
Traditionally a study is said to be "statistically significant" if the odds are only 1 in 20 that the result could be pure chance. But in a complicated field where there are many potential hypotheses to sift through - such as whether a particular gene influences a particular disease - it is easy to reach false conclusions using this standard. If you test 20 false hypotheses, one of them is likely to show up as true, on average.
Odds get even worse for studies that are too small, studies that find small effects (for example, a drug that works for only 10% of patients), or studies where the protocol and endpoints are poorly defined, allowing researchers to massage their conclusions after the fact.
Surprisingly, Ioannidis says another predictor of false findings is if a field is "hot", with many teams feeling pressure to beat the others to statistically significant findings.
But Solomon Snyder, senior editor at the Proceedings of the National Academy of Sciences, and a neuroscientist at Johns Hopkins Medical School in Baltimore, US, says most working scientists understand the limitations of published research.
"When I read the literature, I'm not reading it to find proof like a textbook. I'm reading to get ideas. So even if something is wrong with the paper, if they have the kernel of a novel idea, that's something to think about," he says.
The claim is that a little more than 50% of all scientific papers are wrong (so that half of them are right, no?), not that each and every paper is 50% wrong in its contents. That WOULD be a major disaster, lol!
BTW, when I was a grad student in statistics we read an article in one of the stat journals that said that 80% of all the articles published in scientific journals have insufficient "power" (the probability you'll reject the null hypothesis if it's wrong). Talk about shooting yourself in the foot! Here the researcher spends years and years collecting data trying to demonstrate some earth-shaking effect, and yet he designs his own study to have only the slimmest chance of detecting the effect, EVEN IF IT EXISTS! And yet these papers still managed to get published, claiming they have enough evidence to reject the null hypothesis (the skeptic's hypothesis). That's telling us something, right there.
lol
The publication standards are 3-sigma and 5-sigma, as I said. I spelled out their values (approximately) just because I realized that not everyone would know or remember what they meant. In all seriousness, I worried whether I was being too pedantic in explaining them.
I've only been publishing and doing peer review for about 10 years now. The stat software has been available for that entire time. So I really don't have an opinion on whether it has changed. Part of the problem is that the peer-reviewers are, like the authors, usually specialists in the subject matter of the Journal and frequently are not strong in statistics.
The question is whether the subject is relevant to evolution. If the scientific principles behind nuclear weapons works, so must the principles behind radiometric dating. Many papers on evolution use radiometric dating as their primary evidence. Now do you see the connection?
"Consensus science" is always right. If "most scientists" agree, than it must be true. < /sarcasm
How do you expect to get any government grants with crazy talk like that?
Today, I am complaining about evrything. Your reply was the most informative on the subject.
Can we talk about global warming instead?
You're right. It's a quasi-religion based on faith in the 'scientific method'. And like any other religion it has its share of quacks, hustlers, con men and myth makers. However, science used honestly and with an understanding of its limitations is a great benefit to mankind.
The professor kept the data and each following year, she had a new student perform the same test, with the same result. After a few years, her student made an error in calculation and reported the results were significant.
The professor then submitted the study for publication. She included means and standard deviations for the various groups in her study.
Apparently, she was unaware that any reader of the published aricle could verify the statistical test from the means and SDs. One reader dicovered the error and demanded a retraction.
The professor's response? She loudly complained, I should never have included the standard deviations!
By contrast, in physics, findings are never final. Newton's formulation held up for about two centuries until superseded by Einstein's, which will in due course be superseded as well.
But when you get to the so-called soft sciences (social sciences), the record is more dismal. One of the problems is that there is a trade-off between relevance and reliability. You can have a paper with high reliability but no relevance, and you can have a paper with high relevance but no reliability. What you can never have is a paper with high relevance and high reliability.
Nevertheless, the vigor of our economy is predicated on the power of our technology, which is in turn dependent on the quality of our science. Despite all the weaknesses noted on this thread, no one can deny that we ride around in jet planes, eat and drink safe foods, live in relatively safe dwellings, and trade these messages over a network, using computers and software, all of which are the products of science, from which we can conclude that at least some of the products of science are usable.
Assuming, as good scientists do, that there will always be more to learn about most subjects, it is technically accurate to say that all statements about scientific subjects are wrong or at least incomplete. That being said, scientists and engineers who put scientific findings into practice are pretty much right about most things.
This is not surprising in the least.
Yes, I do see. However, the fact that the principles behind something are correct does not mean that the experimental structure or the interpretation of data is correct.
I think I'd rather just blow things up.
I thought it was something like that. Sometimes I get confused.
read later bump
I like the scientific method. It's an invaluable tool in my every day life but like any tool it can be abused.
Act locally, bludgeon globally, as they are fond of doing.
I just think modern science had institutionalized flim flam. Nothing wrong with it, I guess, if one can get away with it. Just don't expect me to believe anything they say just because they're published in JAMA or whatever.
To thine own self be true...or relatively true.
Maybe the real reason "scientists" despise "psychics" is because the "psychics" cut into the "scientists" margins?
Just a wild speculation, at least until I can fabricate some data and get it published in Science.
Was this in the social sciences? Her whining sounds like she would support outcome based education and self-esteem.
Well, it depends on what is "right." Most real scientists don't claim to know the truth, only a reasonable approximation of it.
The only people who are RIGHT generally run kooky websites and live in momma's basement. :P
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.