Peer Review Needs Improvement ('peer review' is often equated with 'gold standard'. Far from it)

American Thinker ^ | 5/2/2009 | Jack Dini

[SeekAndFind]

Scientific issues often deal with items that are too complex and technical for many people to grasp. When these issues touch on public policy, the problem of educating the public is compounded because partisan special-interest groups often mischaracterize or disregard the best available evidence in order to further their agendas. Obviously, this can often result in the enactment of ineffective or counterproductive laws or government programs and also unnecessarily excite the public.

A National Academy of Engineering report says, "As a society, we are not even fully aware of or conversant with the technologies we use every day. In short, we are not ‘technologically literate.'" (1)

Peer review is supposed to help in this area. What is it?

"Peer review is the process by which research and scholarship are evaluated by other experts in one's field. The depth and breadth of formal peer review varies by field; for example, the number of reviewers, as well as whether or not they and article authors will remain anonymous to one another, differs across science, social science, and humanities disciplines. Informal peer review also takes place after research results are published in an article; others in the field weigh in with observations and experiences that question, critique, or support the authors' assertions," reports Leslie Madsen. (2)

Yet, as John Moore writes in Nature,

"It's been peer reviewed so it must be right, right? Wrong! Not everything in the peer-reviewed literature is correct. Indeed, some of it is downright bad science. Professional scientists usually know how to rate papers within their own fields of expertise (all too often very narrow ones nowadays). We realize that some journals are more stringent than others in what they will accept, and that peer review standards can unfortunately be too flexible. A lust for profits has arguably led to the appearance of too many journals, and so it can be all too easy to find somewhere that will publish poor-quality work." (3)

Peer reviewers merely give advice to the editor as to whether a paper should be published. There is no warranty that the results are correct or that they can be reproduced. The reviewers whom the editor picks say yes, we would like to see this in the journal; that is the start and finish of it. (4)

Yet, the term ‘peer review' is often equated with ‘gold standard'. Hence, the politically motivated, lazy or unscrupulous can use the peer-reviewed literature selectively to make arguments that are seriously flawed, or even damaging to public policy. Chris Mooney, in The Republican War on Science (Basic Books, 2005) provides several examples of how this operates in the political world. (3)

Ragner Levi observes,

"As every experienced medical reporter knows, there is no guarantee that published studies are reliable-far from it. Though peer reviewed articles are generally less biased than non-reviewed articles, even highly unreliable results can be published in peer reviewed scientific journals. In a randomized controlled trial looking at peer review, a paper with eight significant errors was sent to 420 reviewers in JAMA's database. None spotted more than five errors, and most not more than two." (5)

Here's another example. Another report in JAMA found that one-third of studies published in three reputable peer reviewed journals didn't hold up. John Ioannidis looked at 45 studies published between 1990 and 2003 and found that subsequent research contradicted the results of seven of those studies, and another seven were found to have weaker results than originally published. In other words, 32% did not withstand the test of time.

This translates into a lot of medical misinformation! Ioannidis reviewed high-impact journals including The New England Journal of Medicine, The Journal of the American Medical Association (JAMA) and Lancet, along with a number of others. Each article had been cited at least 1,000 times, all within a span of 13 years. In a number of cases, the explanation for the discrepancies was in precisely what you'd suspect, sample size. The smaller the group, the shorter the study, the more likely it was that subsequent, deeper investigation contradicted or altered the original thesis. Where was the peer review? (6)

Garrett Lisi notes,

"This old system persists because academic career development often depends on which journals scientists can get their papers into, and it comes at a high cost-in money, time and stress. I think a better peer review system could evolve from reviewers with good reputations picking the papers they find interesting out of an open pool, and commenting on them. (7)

There is hope for improvement. In his book The Great Betrayal, Horace Freeland Judsonb agrees there is a distressing downward spiral in the peer review system, but holds out hope that ‘open review,' which prevent reviewers from hiding behind anonymity, and open publication on the Internet rather than in peer reviewed journals, may solve some of the problems. (8)

Notes

1. Technically Speaking: Why All Americans Need to Know More About Technology, G. Pearson and A. T. Young, Editors, (Washington, DC, National Academy Press, 2002), 1

2. Leslie Madsen Brooks, www.blogher.com/peer-review-science-it-broken, March 14, 2009

3. John Moore, "Perspective: Does peer review mean the same to the public as it does to scientists?," Nature, (2006), doi: 10.1038/nature05009

4. Stephen McIntyre and Ross McKitrick, "The Hockey Stick Debate: Lessons in Disclosure and Due Diligence," (Washington, DC, George Marshall Institute, May 11, 2005)

5. Ragner Levi, Medical Journalism, (Ames, Iowa, Iowa State University, 2001), 64

6. John P. A. Ioannidis, "Contradicted and Initially Stronger Effects in Highly Cited Clinical Research," JAMA, 294 (2), 218, July 13, 2005

7. Greg Boustead, "Garrett Lisi's Exceptional Approach to Everything," seedmagazine.com, November 17, 2008

8. Horace Freeland Judson, The Great Betrayal: Fraud in Science, (New York, Harcourt, Inc., 2004)

[Kansas58]

“peer review” is a way for the left to talk down to conservatives.

THEIR “studies” have been “peer reviewed” so THEY must be correct and WE must be wrong!

[SeekAndFind]

THEIR “studies” have been “peer reviewed”

But who are these "peers' and more importantly, what was the substance of the review ?

[Always Right]

Studies that scientist agree with are rubbered stamped without any scrutiny, while studies that go against the politically correct dogma are torn apart with every lame argument they can come up with. This is particularly true on global warming.

[jdege]

Generally speaking, the standard for peer review in the medical journals is only a weak shadow of that for the physical sciences.

[wendy1946]

A normal person wishing to understand what has become of academia in recent years, has several starting points, including of course DeSouza's "Illiberal Education", Bloom's "Closing of the American Mind", and a far more intense and all-encompassing book in the Quirck/Bridwell book "Abandoned: The Betrayal of the American Middle Class Since WW-II".

Martin Anderson's "Imposters in the Temple" is another item to add to that little list.

The basic job of colleges and universities should be teaching. There still are schools at which that holds true, but they are an exception at present. Anderson describes the trivial pursuits which have replaced teaching as the major objective of many if not most professors, in many if not most schools:

"For most professors, the surest route to scholarly fame (and some fortune) is to publish in the distinguished academic journals of their field. Not books, or treatises, for these are rare indeed, but short, densely packed articles of a dozen pages or so.

"The successful professor's resume will be littered with citations of short, scholarly articles, their value rising with the prestige of the journal. These studious articles are the coin of the realm in the academic world. They are the professor's ticker to promotion, higher salary, generous research grants, lower teaching loads, and even more opulent office space.

"...These are supposed to be scholarly pieces, at the cutting edge of new knowledge.

"But now I must confess something. Many years ago when I read these articles regularly as part of my academic training and during my early years as a professor, I was bothered by the fact that I often failed to find the point of these articles, even after wading through the web of jargon, mathematical equations, and turgid English. Perhaps when I get older and wiser I will appreciate them more, I thought. Well, I am now fifty-five years old, and the significance of most academic writing continues to elude me."

"In recent years, I have conducted an informal survey. Whenever the opportunity presents itself, I ask scholars about their academic journal reading habits. For example, I recently asked a colleague, a man with a solid reputation as a scholar, what he considered to be the most important academic journal in his field of study. An economist, he immediately replied "The American Economic Review".

"Let me ask you a question", I said. "Take, say, all of the issues of the last five years. What is your favorite article?"

"...Sure enough, he answered like all the rest. There was a silence of a few seconds, and then he cleared his throat a bit and, looking somewhat guilty and embarassed, said "Well, I haven't been reading it much lately." When pressed, he admitted that he could not name a single article which he had read during the last five years which he found memorable. In fact, he probably had not read any articles, but was loath to admit it.

"...There are exceptions of course, a handful of men and women in every field who do read these articles and try to comprehend any glimmers of meaning or significance they might contain. But, as a general rule, nobody reads the articles in academic journals anymore.

"...There is a mystery here. For while these academic publications pile up, largely unread, on the shelves of university libraries, their importance to a professor's career continues unabated. Scarcely anyone questions these proofs of erudition on a resume.

"...One reason why these research articles are automatically accepted as significant and important is that they have survived the criticism of "peer review" before being published.

"...Some of the manuscript reviews are done 'blind', with the author's name stripped off, while others are not and the reviwer knows exactly whom he or she is evaluating. Given what is at stake in peer reviewing... it would not be unreasonable to worry a little about corruption sneaking in.

"But these questions are not explored. The fact that some fields of study are small enough that the intellectuals involved in them are all known to eachother, or that friends review friends, or that reviewers repay those who reviewed their own writings favorably in the past -- all these potential problems are ignored...

Anderson, of course, is describing a sort of a ritualized and formalized version of what college frats sometimes refer to as a "circle jerk".

[freespirited]

“Peer reviewed” means higher quality than the crap you find in the MSM.

It doesnt say much, but something is still better than nuttin’.

[SeekAndFind]

Generally speaking, the standard for peer review in the medical journals is only a weak shadow of that for the physical sciences.

I am also very curious as to how the Nobel Peace Prize committee actually peer reviewed Al Gore's powerpoints before giving him the Nobel Prize.

Did they actually consult scientists ( including those who disagree ) to determine if his alarmism has any factual basis ?

[GodGunsGuts]

Related:

Creationism, science and peer review

http://creation.com/images/pdfs/tj/j22_1/j22_1_44-49.pdf

[A Strict Constructionist]

“Generally speaking, the standard for peer review in the medical journals is only a weak shadow of that for the physical sciences.”

If the paper comes from certain institutions or researchers it is accepted, how dare we disagree with one of the annointed. Check out where the NIH fraud comes from. The more liberal the institution the more likely the fraud in my opinion.

[Jaime2099]

This is a great post thank you.

Not too long ago someone posted a peer reviewed article to prove their point about evolution and I was astonished at how much biased, full of speculation, and downright guessing was in the paper. Scientific peer review has become a write what we like and you're approved community. This is a very bad trend for the scientific community, one I hope is corrected promptly.

[neverdem]

BTTT!

[PapaBear3625]

Studies that scientist agree with are rubbered stamped without any scrutiny, while studies that go against the politically correct dogma are torn apart with every lame argument they can come up with. This is particularly true on global warming.

Global warming, in particular, points to a weakness in the peer review process. The only people qualified to review a global warming study are scientists who themselves are involved in global warming studies, and whose funding is dependent upon global warming continuing to be considered a crisis.

Science is now just another kind of prostitution. The scientist is in the business of getting funding grants, and his ability to continue getting grants is dependent upon him giving the results that grant issuers approve of. A peer reviewer risks his own funding if he gives a bad review to a study which his funders approve of.

[PapaBear3625]

A version of “peer review” that I think would be an improvement: forget the journals, and post the study on the researcher’s blog. Invite comments. The researcher may NOT edit or delete comments. After some amount of time sufficient for a reasonable amount of commenting, the researcher may withdraw the article or leave it up.

[r-q-tek86]

ping

[Always Right]

Science is now just another kind of prostitution.

Yep, not much different. There are some people who will defend the scientific community, but overall I am disgusted by what science puts its name on. I am only an engineer by degree though, so what do i know.

[WondrousCreation]

“Peer review” is nothing more than total oligarchical tyranny.

What we get is a bunch of closed-minded half-educated sycophants to secularism proudly wearing their “PhD” from godless liberal institutions refusing to hear any new idea that could advance science out of the dead-standstill rut it has fallen into over the last several decades.

-Dare suggest that basic physical principles like thermodynamics prohibit the forward evolution of new structures and functions in life? Blackballed for life.

-Dare suggest that evidence of mass extinction and rapid sedimentation are evidence for a global flood? Blackballed for life.

-Dare suggest that calibration of radiometric dating by fossils and physical constants that are shown to change with time renders the method unreliable, and that the data better support a young earth? Blackballed for life.

-Dare suggest that the astronomical redshift is better suited to an anthropocentric unvierse based on hydrogenic white-earth cosmology than a “Big Bang” that violates gravitational and energy conservation laws? Blackballed for life.

-Dare suggest that vaccines are poisoning our children with toxins, that natural remedies can provide better cures for cancer than invasive surgery, or that supposed “unstoppable plagues” may indeed just be harmless retroviruses? Blackballed for life.

-Dare suggest that nature isn’t truly random and that quantum fuzziness models that contradict common sense can be improved upon by using modified models of the atom and relying on classical mechanics and electromagnetism? Blackballed for life.

-Dare suggest that earth-centered coordinate systems are better suited for many astronomical calculations, or that non-coordinate-specific general relativity provides insights into the possible nature of a universal “center”? Blackballed for life.

-Dare suggest that electromagnetism, which is billions of times stronger than gravity, as observed many times on earth, is the true unifying binding force of the universe? Blackballed for life.

-Dare suggest that the immense data processing required to support the myth of global warming might actually be a sham? Blackballed for life.

-Dare suggest that homosexuality is a personal choice rather than a hereditary condition (which contradicts the Darwinian principle they are so endeared to)? Blackballed for life.

Peer review has become nothing more than a tool for the self-appointed elite to prop up their own pet theories while rejecting new insights that might make them compete for their livelihoods, or even more commonly, just might be too deep for their puny minds to comprehend.

All modern “peer review” in science has done is deprive the scientific community of a unity with the Creator, and the trickle-down effect has pushed a once great nation away from the Almighty.

We have some great conservative minds here on FreeRepublic - through a rich spectrum of online activism, hopefully we can supplement the peer review process with good old common sense criticism to tear holes through the flimsy charade that modern science has become, and restore unity between science and its true source, the Creator.

[ColdWater]

Nice summary showing the stupidity of Dare!

[Always Right]

Great post

[SunkenCiv]

Somebody back then had the bright idea that, if peer reviewers were anonymous and free from accountability, they would be more candid and more truthful. For about five decades, ...NSF, NASA and other agencies have been doing what no foreign adversary or terrorist organization has been able to do: They have been slowly and imperceptibly undermining American science, driving America toward third-world status in science. Secret, unaccountable reviews - frequently by one's competitors - give unfair advantage to reviewers who would falsely berate a competitor's proposal for research funds... The system has been to open to corruption for decades, and remains open to further corruption... There is a far, far more devastating consequence of secret, unaccountable reviews: Out of fear of being "denounced" in secret reviews, many scientists have become pale-gray, defensive, adopting only the consensus-approved viewpoint and refraining from discussing anything that might be considered a challenge to other's work or to the funding agency's programs. Political correctness is the order. -- Alvarez by Luis Alvarez (page 184)

I must reiterate my feeling that experimentalists always welcome the suggestions of the theorists. But the present situation is ridiculous... In my considered opinion the peer review system, in which proposals rather than proposers are reviewed, is the greatest disaster to be visited upon the scientific community in this century. No group of peers would have approved my building the 72-inch bubble chamber. Even Ernest Lawrence told me that he thought I was making a big mistake. He supported me because my track record was good. I believe U.S. science could recover from the stultifying effects of decades of misguided peer reviewing if we returned to the tried-and-true method of evaluating experimenters rather than experimental proposals. Many people will say that my ideas are elitist, and I certainly agree. The alternative is the egalitarianism that we now practice and that I've seen nearly kill basic science in the USSR and in the People's Republic of China. -- ibid (pp 200-201)

Thanks neverdem.

· join · view topics · view or post blog · bookmark · post new topic ·