Posted on 08/30/2005 10:29:44 AM PDT by LibWhacker
Most published scientific research papers are wrong, according to a new analysis. Assuming that the new paper is itself correct, problems with experimental and statistical methods mean that there is less than a 50% chance that the results of any randomly chosen scientific paper are true.
John Ioannidis, an epidemiologist at the University of Ioannina School of Medicine in Greece, says that small sample sizes, poor study design, researcher bias, and selective reporting and other problems combine to make most research findings false. But even large, well-designed studies are not always right, meaning that scientists and the public have to be wary of reported findings.
"We should accept that most research findings will be refuted. Some will be replicated and validated. The replication process is more important than the first discovery," Ioannidis says.
In the paper, Ioannidis does not show that any particular findings are false. Instead, he shows statistically how the many obstacles to getting research findings right combine to make most published research wrong.
Massaged conclusions
Traditionally a study is said to be "statistically significant" if the odds are only 1 in 20 that the result could be pure chance. But in a complicated field where there are many potential hypotheses to sift through - such as whether a particular gene influences a particular disease - it is easy to reach false conclusions using this standard. If you test 20 false hypotheses, one of them is likely to show up as true, on average.
Odds get even worse for studies that are too small, studies that find small effects (for example, a drug that works for only 10% of patients), or studies where the protocol and endpoints are poorly defined, allowing researchers to massage their conclusions after the fact.
Surprisingly, Ioannidis says another predictor of false findings is if a field is "hot", with many teams feeling pressure to beat the others to statistically significant findings.
But Solomon Snyder, senior editor at the Proceedings of the National Academy of Sciences, and a neuroscientist at Johns Hopkins Medical School in Baltimore, US, says most working scientists understand the limitations of published research.
"When I read the literature, I'm not reading it to find proof like a textbook. I'm reading to get ideas. So even if something is wrong with the paper, if they have the kernel of a novel idea, that's something to think about," he says.
It has always been important to have reproducible results.
Well, duh.
When a person, scientist or not, sets out to prove or disprove a theory, they saddle up with the bias of their intent.
Searching for the truth is a completely different thing.
The problem in science is, despite all the talk of replication, that replication doesn't get you anything. Replication is considered low. Only "first discovery" is rewarded with more grant money. There is no incentive to try and repeat another's experiment for verification.
It was bound to happen-a study about studies. It's really just common sense. If you notice how much of what passes itself off as 'science' just happens to validate some leftist group's political agenda, like global warming, you start to wonder about the validity of the study. Political correctness is torturing history exactly the same way. The earth is flat and don't you dare question it. Especially on a college campus.
"Science-as-settled-fact" ping
Most income tax returns are "wrong" too. What's the articles point?
LOL!
I believe the point is... don't believe everything you read.
But I thought "science" was always right!
In physics, the standard is 3-sigma (about a chance in 400) to claim evidence for something, and 5-sigma (about a chance in 1.8 million) to claim discovery. I'm constantly appalled at the weak statistical cases that are bandied about in the press as the gospel truth, particularly when it comes to medical studies.
Science can handle a poor signal-to-noise ratio, but public policy cannot.
Only in philosophy where there never is a check with reality can one bad idea build on another without ever getting corrected.
I would include education also.
Ideally this is true, but it is extremely rare to find a paper published that directly and specifically refutes another.
And when you do find this, often the first paper is in a "high class" journal but the refutation will only get published in a "low class" journal. Which means that often the refutation is ignored.
After a few decades of experience with research in psychology, like the commentator, I don't take any published study on a stand-alone basis, but accept things, if ever, only after they have been replicated by the researcher's opponent.
Even if several studies are reported in the media with a certain finding, they are not trustworthy because one remarkable result generates a bandwagon for uncritical acceptance of further findings of the same sort.
There is no way to sort out the truth, but only to enlarge the context we have for understanding things.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.