Free Republic
Browse · Search
News/Activism
Topics · Post Article

Skip to comments.

Most scientific papers are probably wrong
New Scientist ^ | 8/30/05 | Kurt Kleiner

Posted on 08/30/2005 10:29:44 AM PDT by LibWhacker

Most published scientific research papers are wrong, according to a new analysis. Assuming that the new paper is itself correct, problems with experimental and statistical methods mean that there is less than a 50% chance that the results of any randomly chosen scientific paper are true.

John Ioannidis, an epidemiologist at the University of Ioannina School of Medicine in Greece, says that small sample sizes, poor study design, researcher bias, and selective reporting and other problems combine to make most research findings false. But even large, well-designed studies are not always right, meaning that scientists and the public have to be wary of reported findings.

"We should accept that most research findings will be refuted. Some will be replicated and validated. The replication process is more important than the first discovery," Ioannidis says.

In the paper, Ioannidis does not show that any particular findings are false. Instead, he shows statistically how the many obstacles to getting research findings right combine to make most published research wrong.

Massaged conclusions

Traditionally a study is said to be "statistically significant" if the odds are only 1 in 20 that the result could be pure chance. But in a complicated field where there are many potential hypotheses to sift through - such as whether a particular gene influences a particular disease - it is easy to reach false conclusions using this standard. If you test 20 false hypotheses, one of them is likely to show up as true, on average.

Odds get even worse for studies that are too small, studies that find small effects (for example, a drug that works for only 10% of patients), or studies where the protocol and endpoints are poorly defined, allowing researchers to massage their conclusions after the fact.

Surprisingly, Ioannidis says another predictor of false findings is if a field is "hot", with many teams feeling pressure to beat the others to statistically significant findings.

But Solomon Snyder, senior editor at the Proceedings of the National Academy of Sciences, and a neuroscientist at Johns Hopkins Medical School in Baltimore, US, says most working scientists understand the limitations of published research.

"When I read the literature, I'm not reading it to find proof like a textbook. I'm reading to get ideas. So even if something is wrong with the paper, if they have the kernel of a novel idea, that's something to think about," he says.


TOPICS: News/Current Events
KEYWORDS: bias; conclusions; creationping; data; massaged; papers; scientific; statistics; wodlist; wrong
Navigation: use the links below to view more comments.
first 1-5051-100101-150151-158 next last

1 posted on 08/30/2005 10:29:45 AM PDT by LibWhacker
[ Post Reply | Private Reply | View Replies]

To: LibWhacker
I don't believe this article.
2 posted on 08/30/2005 10:33:12 AM PDT by inquest (FTAA delenda est)
[ Post Reply | Private Reply | To 1 | View Replies]

To: LibWhacker

It has always been important to have reproducible results.


3 posted on 08/30/2005 10:33:58 AM PDT by knuthom
[ Post Reply | Private Reply | To 1 | View Replies]

To: LibWhacker
Most scientific papers are probably wrong...

Well, duh.

When a person, scientist or not, sets out to prove or disprove a theory, they saddle up with the bias of their intent.

Searching for the truth is a completely different thing.

4 posted on 08/30/2005 10:34:25 AM PDT by Dark Skies ("The sleeper must awaken!")
[ Post Reply | Private Reply | To 1 | View Replies]

To: LibWhacker
Some will be replicated and validated. The replication process is more important than the first discovery," Ioannidis says.

The problem in science is, despite all the talk of replication, that replication doesn't get you anything. Replication is considered low. Only "first discovery" is rewarded with more grant money. There is no incentive to try and repeat another's experiment for verification.

5 posted on 08/30/2005 10:34:39 AM PDT by Rokurota (.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: LibWhacker
Assuming that the new paper is itself correct, problems with experimental and statistical methods mean that there is less than a 50% chance that the results of any randomly chosen scientific paper are true.

I am left to wonder what statistical method the author used. :-)
6 posted on 08/30/2005 10:37:36 AM PDT by ARCADIA (Abuse of power comes as no surprise)
[ Post Reply | Private Reply | To 1 | View Replies]

To: LibWhacker

It was bound to happen-a study about studies. It's really just common sense. If you notice how much of what passes itself off as 'science' just happens to validate some leftist group's political agenda, like global warming, you start to wonder about the validity of the study. Political correctness is torturing history exactly the same way. The earth is flat and don't you dare question it. Especially on a college campus.


7 posted on 08/30/2005 10:39:02 AM PDT by Spok (Est omnis de civilitate.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: LibWhacker
Alrigty then, ditch science. We know how well it worked without it in the DARK AGES! </sarc>
8 posted on 08/30/2005 10:39:03 AM PDT by CarrotAndStick (The articles posted by me needn't necessarily reflect my opinion.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: DaveLoneRanger

"Science-as-settled-fact" ping


9 posted on 08/30/2005 10:39:35 AM PDT by My2Cents ("It takes a nation of candyasses to hold this military back.")
[ Post Reply | Private Reply | To 1 | View Replies]

Comment #10 Removed by Moderator

To: LibWhacker

Most income tax returns are "wrong" too. What's the articles point?


11 posted on 08/30/2005 10:42:06 AM PDT by narby (There are Bloggers, and then there are Freepers.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: LibWhacker; gobucks; mikeus_maximus; MeanWestTexan; JudyB1938; isaiah55version11_0; bondserv; ...
(((Creationist Ping)))



You have been pinged because of your interest in matters of Creation vs. Evolution, Creation trumping evolution, and evolutionary fraud. Freep-mail me if you want on/off this list.

Colossians 1:16 "For by him all things were created: things in heaven and on earth, visible and invisible, whether thrones or powers or rulers or authorities; all things were created by him and for him."



Do not assume this is true...just remember that everything we are told by the governing science elites who know so much more than we could possibly fathom should be taken with a grain of salt. Just remember it when you see evolutionists post huge reams of spam links from sophisticated scientific journals...just because they have PhD doesn't mean they're infallible.
12 posted on 08/30/2005 10:43:08 AM PDT by DaveLoneRanger (Be on your guard; stand firm in the faith; be men of courage; be strong. (1 Corinthians 16:13))
[ Post Reply | Private Reply | To 1 | View Replies]

To: inquest

LOL!


13 posted on 08/30/2005 10:43:30 AM PDT by <1/1,000,000th%
[ Post Reply | Private Reply | To 2 | View Replies]

To: narby
What's the articles point?

I believe the point is... don't believe everything you read.

14 posted on 08/30/2005 10:43:43 AM PDT by rhombus
[ Post Reply | Private Reply | To 11 | View Replies]

To: TaxRelief

But I thought "science" was always right!


15 posted on 08/30/2005 10:44:12 AM PDT by Tax-chick (Oklahoma is the cultural center of the universe ... take me back to Tulsa!)
[ Post Reply | Private Reply | To 1 | View Replies]

To: LibWhacker
Traditionally a study is said to be "statistically significant" if the odds are only 1 in 20 that the result could be pure chance.

In physics, the standard is 3-sigma (about a chance in 400) to claim evidence for something, and 5-sigma (about a chance in 1.8 million) to claim discovery. I'm constantly appalled at the weak statistical cases that are bandied about in the press as the gospel truth, particularly when it comes to medical studies.

Science can handle a poor signal-to-noise ratio, but public policy cannot.

16 posted on 08/30/2005 10:44:13 AM PDT by Physicist
[ Post Reply | Private Reply | To 1 | View Replies]

To: bobbdobbs

Only in philosophy where there never is a check with reality can one bad idea build on another without ever getting corrected.

I would include education also.


17 posted on 08/30/2005 10:44:23 AM PDT by PeterPrinciple (Seeking the truth here folks.)
[ Post Reply | Private Reply | To 10 | View Replies]

To: ARCADIA
I think the article refers to less than 50% of the papers providing conclusions stand the test of time, and or rigorous open peer review.

Results that are not reproducible are not to be believed, a classic is global warming where we have one study, the Mann study, that all the libs like to latch onto to prove man caused global warming, while at the same time denying respectability to contrary views.
18 posted on 08/30/2005 10:44:31 AM PDT by Tarpon
[ Post Reply | Private Reply | To 6 | View Replies]

To: bobbdobbs
Most scientists regard published results with scepticism. And of course any follow on work based on flawed papers will come up with bad results.

Ideally this is true, but it is extremely rare to find a paper published that directly and specifically refutes another.

And when you do find this, often the first paper is in a "high class" journal but the refutation will only get published in a "low class" journal. Which means that often the refutation is ignored.

19 posted on 08/30/2005 10:44:34 AM PDT by Rokurota (.)
[ Post Reply | Private Reply | To 10 | View Replies]

To: LibWhacker
Most scientific papers are probably wrong

After a few decades of experience with research in psychology, like the commentator, I don't take any published study on a stand-alone basis, but accept things, if ever, only after they have been replicated by the researcher's opponent.

Even if several studies are reported in the media with a certain finding, they are not trustworthy because one remarkable result generates a bandwagon for uncritical acceptance of further findings of the same sort.

There is no way to sort out the truth, but only to enlarge the context we have for understanding things.

20 posted on 08/30/2005 10:44:53 AM PDT by Marylander
[ Post Reply | Private Reply | To 1 | View Replies]

To: narby

Heck, I'd be happy if just the NIH would stop throwing taxpayer money at poorly conceived and executed studies.


21 posted on 08/30/2005 10:46:27 AM PDT by LibWhacker
[ Post Reply | Private Reply | To 11 | View Replies]

To: LibWhacker
"When I read the literature, I'm not reading it to find proof like a textbook. I'm reading to get ideas. So even if something is wrong with the paper, if they have the kernel of a novel idea, that's something to think about," he says.

Speaking as someone who has served his time in a research lab in graduate school, I can testify to the truth to this. As a general rule, nobody reads journals for any kind of scientific proof (occasionally a paper comes along that really does definitively prove something, but it's pretty rare). We read them to get ideas, and to gather a general feel for what other researchers are interested in.

Scientists are generally aware of one another's biases. When you read a scientific paper, it's usually pretty darn clear what the scientist was trying to show - what conclusion the scientist would have liked to see. If you read these papers as a trained scientist, you learn that their real value lies not in the researcher's outcomes, but in what the researcher had to go through in order to produce those results! That is how you actually learn anything about reality when reading scientific papers.

22 posted on 08/30/2005 10:46:32 AM PDT by Omedalus
[ Post Reply | Private Reply | To 1 | View Replies]

To: Spok

A good friend of mine used to work as a lab tech for animal experiments, or as he puts it, he "tortured animals for a living."

He tells the most appalling stories about how studies were designed to avoid finding what they didn't want to find.

The phenomenon you describe is unfortunately not at all limited to those with a leftist agenda. Money can also do a good job of motivating a scientist to find a way to find what he, or his backers, want found.


23 posted on 08/30/2005 10:47:37 AM PDT by Restorer (Liberalism: the auto-immune disease of democracies.)
[ Post Reply | Private Reply | To 7 | View Replies]

To: inquest
I don't believe this article.

You should. The statistics they cite are unexceptional with respect to the multiple hypotheses problem (eg if you test 20 false hypotheses at the 95% confidence level), one of them will probably test as true at the 95% level simply as a matter of random noise.

For example, suppose you are testing the output of DNA chips as a preditor of diabetes. There are some 2,000 hypotheses per experiment. That is, the chip produces 2,000 measurements for, say, 2,000 genes, and then those measurements are correlated with a disease state. At the 95% confidence level, you are bound to get many many correlations that seem significant that are only noise.

I have done a lot of scientific publication, refereed journals and conferences, and serve as the editor of peer-reviewed journal. Many, many first published results are wrong. Not just because of the noise issue. Scientists are folks like everyone else and they will ignore the negative findings and report only the positive. They will not define their endpoint in advance but keep going when it looks like they are almost there and so on. These are statistical no no's but they happen all the time.

Some years ago, I found an error in a paper I had previously written with another author--we were measuring (and reported) a bug in our software, not a real phenomenon. He was the head of a prominent dept in a prominent university. He argued 'til he was blue in the face that we should not report that we had been mistaken.

The system tends to be self-correcting because other folks try to replicate the wrong results and cannot. That gets reported. So eventually the noise is filtered out.

24 posted on 08/30/2005 10:48:15 AM PDT by ModelBreaker
[ Post Reply | Private Reply | To 2 | View Replies]

To: DaveLoneRanger
Just remember it when you see evolutionists post huge reams of spam links from sophisticated scientific journals...just because they have PhD doesn't mean they're infallible.

The article says that most scientific articles are wrong. I don't know why anyone would find that controversial, or even surprising.

For creationism to hold sway, however, they'd essentially all have to be wrong.

25 posted on 08/30/2005 10:48:15 AM PDT by Physicist
[ Post Reply | Private Reply | To 12 | View Replies]

To: ModelBreaker
I figured someone wouldn't get the joke ;-)
26 posted on 08/30/2005 10:48:59 AM PDT by inquest (FTAA delenda est)
[ Post Reply | Private Reply | To 24 | View Replies]

To: LibWhacker
It turned out to be pretty close especially in enumerating the kinds of objects that are visible to the user.

WOW. The very pinnacle of bad science writing. What the headline really should say is that most experiment designs, even those resulting in peer-reviewed publications, are tendentious crap put together by sub-competent "scientists."

Revoking accreditations at some PhD programs seems in order.

27 posted on 08/30/2005 10:50:09 AM PDT by Spike Spiegel
[ Post Reply | Private Reply | To 1 | View Replies]

To: Tax-chick

This is statistics, not science! Science deals with absolute experimentally reproducible phenomena. If you drop something, it will accelerate downwards at the same rate everytime.

Medical and sociological research is NOT science. It may or may not include science in measuring phenomena and mathematics to analyzed observed data.

If 1 person in 100 will die from something it does not mean 100 will die 1%. It means 99 people will not die 100% of the time and 1 person will die 100% of the time.

There is a terrible lack of understanding of science and statistics in this world. 99.99% of the media never took advanced science and or actuarial degrees.


28 posted on 08/30/2005 10:51:53 AM PDT by BillM
[ Post Reply | Private Reply | To 15 | View Replies]

To: Tarpon
Results that are not reproducible are not to be believed

That describes the normal scientific process, with all of its wrong turns,frustrating dead ends, and slow but eventual progress.
29 posted on 08/30/2005 10:51:54 AM PDT by ARCADIA (Abuse of power comes as no surprise)
[ Post Reply | Private Reply | To 18 | View Replies]

To: LibWhacker

I believe that half the artictle is true!


30 posted on 08/30/2005 10:51:54 AM PDT by bubman
[ Post Reply | Private Reply | To 1 | View Replies]

To: LibWhacker

Ioannina family BUMP!!!!!!

Seems very timely given the no pain abortion study,

EVERY SINGLE envirowacko we are doomed study,

EVERY cancer study,

EVERY genes cause X behavior study...


31 posted on 08/30/2005 10:53:20 AM PDT by longtermmemmory (VOTE!)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Rokurota
The problem in science is, despite all the talk of replication, that replication doesn't get you anything. Replication is considered low. Only "first discovery" is rewarded with more grant money. There is no incentive to try and repeat another's experiment for verification.

There is some truth in this. But the factor that keeps it in check is stuff like rivalry, jealously, and wanting to be the guy that knocks off someone else. Plus, a lot of scientists will get a notion that something is wrong and can't let go of it until they satisfy themselves.

Where it doesn't work (in the short term) is in the incredibly sloppy field of computer modeling (global warming being only one small example of sloppy work in that field). There is not usually much emphasis in commercial and academic modeling circles on tying the model to groundtruth. No peer-pressure and ridicule for being wrong means no emphasis on actually being right. An elegant model is cool and significant even if it is completely wrong.

Interestingly, in military modeling circles, the folks there are rabid about tying the models to reality. Maybe because people die when they do not.

32 posted on 08/30/2005 10:54:35 AM PDT by ModelBreaker
[ Post Reply | Private Reply | To 5 | View Replies]

To: LibWhacker

I bet the false results are nearly 100% when it comes to environmental issues.


33 posted on 08/30/2005 10:55:00 AM PDT by Texas_Conservative2
[ Post Reply | Private Reply | To 1 | View Replies]

To: Physicist

Why? Do all scientific articles have something to do with evolution? Surely not everything in physics, chemistry, etc.?


34 posted on 08/30/2005 10:55:54 AM PDT by Tax-chick (Oklahoma is the cultural center of the universe ... take me back to Tulsa!)
[ Post Reply | Private Reply | To 25 | View Replies]

To: BillM
There is a terrible lack of understanding of science and statistics in this world. 99.99% of the media never took advanced science and or actuarial degrees.

I'd wager that more than 99.99% of everyone has no advanced science or actuarial degrees.

35 posted on 08/30/2005 10:57:32 AM PDT by Tax-chick (Oklahoma is the cultural center of the universe ... take me back to Tulsa!)
[ Post Reply | Private Reply | To 28 | View Replies]

To: Rokurota

I would think that money/acclaim does and always has driven research. Scientists are just like all the rest of us: seldom are our motives pure. There are, of course, saints: the men as driven by the desire for natural truth as mystics are for a vision of God.


36 posted on 08/30/2005 10:58:16 AM PDT by RobbyS ( CHIRHO)
[ Post Reply | Private Reply | To 5 | View Replies]

To: LibWhacker
You would have to be naive to think otherwise. If every single paper, out of the tens of thousands that are annually published, or for that matter, even a large percentage were absolutely correct, we would solve all the problems of the universe in a couple of years.

One of the main reason that papers are published are that they are very likely to have some flaws, no matter how minor and need peer review.

37 posted on 08/30/2005 10:58:30 AM PDT by rkhampton
[ Post Reply | Private Reply | To 1 | View Replies]

To: BillM
This is statistics, not science! Science deals with absolute experimentally reproducible phenomena.

Not to quibble, Bill, but a statistical distribution should be reproducible (95% of the time at the 95% confidence level). If, of course, it is measuring something real.

38 posted on 08/30/2005 10:59:44 AM PDT by ModelBreaker
[ Post Reply | Private Reply | To 28 | View Replies]

To: LibWhacker
John Ioannidis, an epidemiologist at the University of Ioannina School of Medicine in Greece, says that small sample sizes, poor study design, researcher bias, and selective reporting and other problems combine to make most research findings false. But even large, well-designed studies are not always right, meaning that scientists and the public have to be wary of reported findings.

Interesting article


Revelation 4:11Intelligent Design
See my profile for info

39 posted on 08/30/2005 10:59:49 AM PDT by wallcrawlr (http://www.bionicear.com)
[ Post Reply | Private Reply | To 1 | View Replies]

To: rhombus
I believe the point is... don't believe everything you read.

I don't believe that....

dang...does that make you right or wrong???

8^)

40 posted on 08/30/2005 11:00:17 AM PDT by The SISU kid (Politicians are like Slinkies. Good for nothing. But you smile when you push them down the stairs)
[ Post Reply | Private Reply | To 14 | View Replies]

To: LibWhacker

If the intent of the article was to "test the audience" or "entertain the audience" I think you've achieved both objectives.


41 posted on 08/30/2005 11:00:27 AM PDT by kipita (Rebel the proletariat response to Aristocracy and Exploitation.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Tax-chick
Why? Do all scientific articles have something to do with evolution? Surely not everything in physics, chemistry, etc.?

Sigh. All in that field of study. But now that you mention it, many papers in those other fields would also have to be systematically wrong.

42 posted on 08/30/2005 11:01:29 AM PDT by Physicist
[ Post Reply | Private Reply | To 34 | View Replies]

To: Dark Skies
When a person, scientist or not, sets out to prove or disprove a theory, they saddle up with the bias of their intent.

Searching for the truth is a completely different thing.

Couldn't have said it better.

To find the truth requires being truthful to ones self

43 posted on 08/30/2005 11:01:34 AM PDT by A message (Only unity will defeat terrorism - do you hear me Democrats? Do you hear me?)
[ Post Reply | Private Reply | To 4 | View Replies]

To: ARCADIA
I don't take any scientific paper or conclusions on face value. Usually the first are rushed to publication, often for the wrong reasons. But once the paper is out it's hard to go against the conclusions in today's world. Science has lost it's way and turned into a political quest for funding.

Skepticism is the best approach.
44 posted on 08/30/2005 11:02:09 AM PDT by Tarpon
[ Post Reply | Private Reply | To 29 | View Replies]

To: LibWhacker
But Solomon Snyder, senior editor at the Proceedings of the National Academy of Sciences, and a neuroscientist at Johns Hopkins Medical School in Baltimore, US, says most working scientists understand the limitations of published research. "When I read the literature, I'm not reading it to find proof like a textbook. I'm reading to get ideas. So even if something is wrong with the paper, if they have the kernel of a novel idea, that's something to think about," he says.

Wow. Just wow. A parallel of "It's the seriousness of the charge that matters!". Liberal leftist socialists are just plain unbelievable. And scary.

45 posted on 08/30/2005 11:04:18 AM PDT by polymuser
[ Post Reply | Private Reply | To 1 | View Replies]

To: LibWhacker

80% of all statistics are made up on the spot.


46 posted on 08/30/2005 11:04:30 AM PDT by SolidRedState (E Pluribus Funk --- (Latin taglines are sooooo cool! Don't ya think?))
[ Post Reply | Private Reply | To 1 | View Replies]

To: inquest
"Most published scientific research papers are wrong, according to a new analysis. Assuming that the new paper is itself correct"

You gotta love it.

47 posted on 08/30/2005 11:04:33 AM PDT by Nathan Zachary
[ Post Reply | Private Reply | To 2 | View Replies]

To: The SISU kid
dang...does that make you right or wrong???

It makes me right with a .05 probability or being wrong (based on a sample size of 1).

48 posted on 08/30/2005 11:05:28 AM PDT by rhombus
[ Post Reply | Private Reply | To 40 | View Replies]

To: Physicist
...particularly when it comes to medical studies...

Oh, doc, you're a man after my own heart, lol! I worked in that field as a statistician helping researchers get published in a "publish or perish" environment. My job? Very often it was just to refute a peer reviewer's criticisms, so that the article could get published. Much to my dismay, my counter-arguments were always accepted and all the articles went on to publication.

That's one reason I'd much rather work with physicists than medical researchers. Not that medical researchers aren't smart, they are very smart, of course. But as a rule, they just don't seem to be as interested in every aspect of their research as you guys are. So the stats get the short shrift from the medicos.

49 posted on 08/30/2005 11:06:08 AM PDT by LibWhacker
[ Post Reply | Private Reply | To 16 | View Replies]

To: BillM
This is statistics, not science! Science deals with absolute experimentally reproducible phenomena. If you drop something, it will accelerate downwards at the same rate everytime.

B-mesons will only decay to D+3pi about 1% of the time, and yet somehow, we call it science.

50 posted on 08/30/2005 11:06:33 AM PDT by Physicist
[ Post Reply | Private Reply | To 28 | View Replies]


Navigation: use the links below to view more comments.
first 1-5051-100101-150151-158 next last

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.

Free Republic
Browse · Search
News/Activism
Topics · Post Article

FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson