Posted on 08/30/2005 10:29:44 AM PDT by LibWhacker
Most published scientific research papers are wrong, according to a new analysis. Assuming that the new paper is itself correct, problems with experimental and statistical methods mean that there is less than a 50% chance that the results of any randomly chosen scientific paper are true.
John Ioannidis, an epidemiologist at the University of Ioannina School of Medicine in Greece, says that small sample sizes, poor study design, researcher bias, and selective reporting and other problems combine to make most research findings false. But even large, well-designed studies are not always right, meaning that scientists and the public have to be wary of reported findings.
"We should accept that most research findings will be refuted. Some will be replicated and validated. The replication process is more important than the first discovery," Ioannidis says.
In the paper, Ioannidis does not show that any particular findings are false. Instead, he shows statistically how the many obstacles to getting research findings right combine to make most published research wrong.
Massaged conclusions
Traditionally a study is said to be "statistically significant" if the odds are only 1 in 20 that the result could be pure chance. But in a complicated field where there are many potential hypotheses to sift through - such as whether a particular gene influences a particular disease - it is easy to reach false conclusions using this standard. If you test 20 false hypotheses, one of them is likely to show up as true, on average.
Odds get even worse for studies that are too small, studies that find small effects (for example, a drug that works for only 10% of patients), or studies where the protocol and endpoints are poorly defined, allowing researchers to massage their conclusions after the fact.
Surprisingly, Ioannidis says another predictor of false findings is if a field is "hot", with many teams feeling pressure to beat the others to statistically significant findings.
But Solomon Snyder, senior editor at the Proceedings of the National Academy of Sciences, and a neuroscientist at Johns Hopkins Medical School in Baltimore, US, says most working scientists understand the limitations of published research.
"When I read the literature, I'm not reading it to find proof like a textbook. I'm reading to get ideas. So even if something is wrong with the paper, if they have the kernel of a novel idea, that's something to think about," he says.
"What's the articles point?
Who knows, maybe everything we believe is wrong.
I suggest that there has never been an instance of a acientific error that was verifiably demonstrated by someone who was outside of the scientific enterprise, using un-scientific methods. [Escape hatch -- I exclude, on grounds of triviality, those occasional cases where a layman finds a fossil or something, and the scientists verify it and perhaps revise their estimates of how widely such a creature may have roamed.]
How do we know this article isn't one of the over 50% that is wrong?
Say it isn't so! An injured guy goes from Ford Field to Ford Hospital in a vehicle whose manufacturer's name is acryonum for "Fix or Repair Daily"? Why, the odds of that must be astronomical! This is clear proof that an advanced but imperceiveable "intelligence" with an inclination toward irony is overseeing the dispatch of rescue vehicles! We must immediately get out our crystal balls and ouija boards to investigate this mystical phenomona!
I recently read someone say something to the effect (I can't remember the exact quote) that virtually all science is just guesswork, until it's put to use, and then it becomes engineering. That seems going almost too far, but I'm wondering if that really isn't a healthy attitude.
Figures!
Say you want to outlaw smoking because you can; stop pretending that there is "science" or "scientific studies" behind the decision, however.
Donald Johanson, is that you?
LOL!
(You should check out an old movie called "Apartment for Peggy" - the heroine makes constant use of fabricated statistics.)
I'm just being a little picky & irritable. Sorry. Part of the problem is we're not all speaking the same language or using the same standards. 3-sigma = 1 chance in 400. How many ways can we creat to say 1 chance in 400.
There is a plethora of contradictory scientific test results on the "market" today. Reminds me of lawyers being able to read anything they want into or out of something. Getting close to babble.
Nope.
The scientist that comes up with a theory to displace evolution will win the Nobel prize and be internationally famous.
If the infamous Discovery Institute were a real scientific endevor, they'd be actually doing science. Rather than pushing lawsuits and press releases.
I'll see if I can find it. Thanks. LOL
But there's a lot of living scum out there...
In some ways the problem is that folks going into tech fields take the mandatory stat class. Then, when they have results to report, they pull out one of the very powerful stat software packages that are available and just plug in their numbers. They never really got a gut sense of what statistics is about. They see it as a legitimizing tool for their real work. So statistics are frequently misused, unintentionally.
The Renaissance and modern science were begun by scholars studying the ancient Greek and Roman texts saved from Byzantium, and re-opened to scholars only a few centuries ago. If ancient texts are relevant, and hold information that can be confirmed, no real scientist could ignore them.
I'm reading books on the development of nuclear weapons right now. Evolution doesn't seem relevant to the subject at all.
But then, I think Oklahoma is the cultural center of the universe :-).
So you'd classify this as a relatively recent problem?
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.