Free Republic
Browse · Search
News/Activism
Topics · Post Article

Skip to comments.

Is Peer Review Broken?
The Scientist ^ | Feb 2006 | ALISON MCCOOK

Posted on 02/14/2006 9:18:41 PM PST by AndrewC

FEATURE
Is Peer Review Broken?
Is Peer Review Broken?

Submissions are up, reviewers are overtaxed, and authors are lodging complaint after complaint about the process at top-tier journals. What's wrong with peer review?

Peter Lawrence, a developmental biologist who is also an editor at the journal Development and former editorial board member at Cell, has been publishing papers in academic journals for 40 years. His first 70 or so papers were "never rejected," he says, but that's all changed. Now, he has significantly more trouble getting articles into the first journal he submits them to.

"The rising [rejections] means an increase in angry authors."
-Drummond Rennie

Lawrence, based at the MRC Laboratory of Molecular Biology at Cambridge, UK, says his earlier papers were always published because he and his colleagues first submitted them to the journals they believed were most appropriate for the work. Now, because of the intense pressure to get into a handful of top journals, instead of sending less-than-groundbreaking work to second- or third-tier journals, more scientists are first sending their work to elite publications, where they often clearly don't belong.

Consequently, across the board, editors at top-tier journals say they are receiving more submissions every year, leading in many cases to more rejections, appeals, and complaints about the system overall. "We reject approximately 6,000 papers per year" before peer review, and submissions are steadily increasing, says Donald Kennedy, editor-in-chief of Science. "There's a lot of potential for complaints."

Everyone, it seems, has a problem with peer review at top-tier journals. The recent discrediting of stem cell work by Woo-Suk Hwang at Seoul National University sparked media debates about the system's failure to detect fraud. Authors, meanwhile, are lodging a range of complaints: Reviewers sabotage papers that compete with their own, strong papers are sent to sister journals to boost their profiles, and editors at commercial journals are too young and invariably make mistakes about which papers to reject or accept (see Truth or Myth?). Still, even senior scientists are reluctant to give speci. c examples of being shortchanged by peer review, worrying that the move could jeopardize their future publications.

So, do those complaints stem from valid concerns, or from the minds of disgruntled scientists who know they need to publish in Science or Nature to advance in their careers? "The rising [rejections] means an increase in angry authors," says Drummond Rennie, deputy editor at Journal of the American Medical Association (JAMA). The timing is right to take a good hard look at peer review, which, says Rennie, is "expensive, difficult, and blamed for everything."

What's wrong with the current system? What could make it better? Does it even work at all?

TOO MANY SUBMISSIONS

Editors at high-impact journals are reporting that the number of submissions is increasing every year (see "Facts and Figures", the table below). Researchers, it seems, want to get their data into a limited number of pages, sometimes taking extra measures to boost their success. Lately, academia seems to place a higher value on the quality of the journals that accept researchers' data, rather than the quality of the data itself. In many countries, scientists are judged by how many papers they have published in top-tier journals; the more publications they rack up, the more funding they receive.

Consequently, Lawrence says he believes more authors are going to desperate measures to get their results accepted by top journals. An increasing number of scientists are spending more time networking with editors, given that "it's quite hard to reject a paper by a friend of yours," says Lawrence. Overworked editors need something flashy to get their attention, and many authors are exaggerating their results, stuffing reports with findings, or stretching implications to human diseases, as those papers often rack up extra references. "I think that's happening more and more," Lawrence says. In fact, in a paper presented at the 2005 International Congress on Peer Review and Biomedical Publication, a prospective review of 1,107 manuscripts submitted to the Annals of Internal Medicine, British Medical Journal (BMJ), and The Lancet in 2003 showed that many major changes to the text demanded by peer review included toning down the manuscript's conclusions and highlighting the paper's limitations. This study suggests that boosting findings may cause more problems by overburdening reviewers even further.

Indeed, sorting through hype can make a reviewer's job at a top journal even more difficult than it already is. At high-impact journals, reviewers need to judge whether a paper belongs in the top one percent of submissions from a particular field - an impossible task, says Hemai Parthasarathy, managing editor at Public Library of Science (PLoS) Biology. Consequently, editors and reviewers sometimes make mistakes, she notes, perhaps publishing something that is really in the top 10%, or passing on a really strong paper. To an outsider, this pattern can look like "noise," where some relatively weak papers are accepted when others aren't, inspiring rejected authors to complain. But, it's an inevitable result of the system, she notes.

THE RELIGION OF PEER REVIEW

Despite a lack of evidence that peer review works, most scientists (by nature a skeptical lot) appear to believe in peer review. It's something that's held "absolutely sacred" in a field where people rarely accept anything with "blind faith," says Richard Smith, former editor of the BMJ and now CEO of UnitedHealth Europe and board member of PLoS. "It's very unscientific, really."

...

more



TOPICS: News/Current Events
KEYWORDS: peerreview; science

1 posted on 02/14/2006 9:18:42 PM PST by AndrewC
[ Post Reply | Private Reply | View Replies]

To: Michael_Michaelangelo; wallcrawlr; DaveLoneRanger; metmom
Ping

Despite the number of complaints lodged at peer review, and the lack of research to show that it works, it remains a valued system, says Rennie. Scientists sigh when they're asked to review a paper, but they get upset if they're not asked, he notes. Reviewing articles is a good exercise, Rennie says, and it enables reviewers to stay abreast of what's going on. Peer review "has many imperfections, but I think it's probably the best system we've got," says Bateson.

Experts also acknowledge that peer review is hardly ever to blame when fraud is published, since thoroughly checking data could take as much time as creating it in the first place.

2 posted on 02/14/2006 9:21:18 PM PST by AndrewC (Darwinian logic -- It is just-so if it is just-so)
[ Post Reply | Private Reply | To 1 | View Replies]

To: AndrewC

Many science/medical journals these days prefer to give space to political harangues, rather than research.


3 posted on 02/14/2006 9:22:42 PM PST by LibFreeOrDie (L'Chaim!)
[ Post Reply | Private Reply | To 1 | View Replies]

To: PatrickHenry

Let's get this party started!


4 posted on 02/14/2006 9:23:09 PM PST by Clemenza (I saw the best minds of my generation destroyed by madness, starving hysterical naked...)
[ Post Reply | Private Reply | To 1 | View Replies]

To: AndrewC

I've been rejected by Science. Who do I whine to?


5 posted on 02/14/2006 9:37:51 PM PST by mlo
[ Post Reply | Private Reply | To 1 | View Replies]

To: mlo
I've been rejected by Science. Who do I whine to?

That's a gimme. Since your submission was not published in a peer-reviewed publication, it is not science.(that is, of course, assuming that it has not been published at all in such publications). So you whine to no one.

6 posted on 02/14/2006 9:43:40 PM PST by AndrewC (Darwinian logic -- It is just-so if it is just-so)
[ Post Reply | Private Reply | To 5 | View Replies]

To: AndrewC

No. Just start a new journal.


7 posted on 02/14/2006 10:17:06 PM PST by ClaireSolt (.)
[ Post Reply | Private Reply | To 6 | View Replies]

To: mlo
Annals of Improbable Research, of course!

Full Disclosure: formerly the Journal of Irreproducible Results... :-)

Try google for ignoble prizes...

Cheers!

8 posted on 02/14/2006 10:38:56 PM PST by grey_whiskers (The opinions are solely those of the author and are subject to change without notice.)
[ Post Reply | Private Reply | To 5 | View Replies]

To: Junior

Archive?


9 posted on 02/15/2006 4:00:38 AM PST by PatrickHenry (Virtual Ignore for trolls, lunatics, dotards, scolds, & incurable ignoramuses.)
[ Post Reply | Private Reply | To 4 | View Replies]

To: AndrewC
If peer review is too highly selective, that isn't a sign of being broken. It's a sign of health.

As for the evidence that peer review works, all you have to do is compare what's in the scientific journals to what's on the internet as a whole.

10 posted on 02/15/2006 4:44:54 AM PST by Physicist
[ Post Reply | Private Reply | To 1 | View Replies]

To: AndrewC
The disgraced Korean stem-cell "scientist" got his impetus from a "peer-reviewed" series of articles in Science magazine. Thanks to such sterling recommendations from such august editor-"scientists", California earmarked $3Billion (that's a B) dollars for embryonic stem cell grants and research.

Looks like that white coat doesn't come with a halo. Maybe scientists put their pants on one leg at a time, like the rest of us.

11 posted on 02/15/2006 5:21:48 AM PST by Mamzelle
[ Post Reply | Private Reply | To 1 | View Replies]

To: AndrewC
Everyone, it seems, has a problem with peer review at top-tier journals.

Peer review is pretty simple these days. If the subject agrees with the politics of the publication, it is blanketly approved. If the subject goes against the politics of the publication, it is blanketly disapproved. The facts are not really scruntinized all that much. It is the sad state of science today.

12 posted on 02/15/2006 5:25:12 AM PST by Always Right
[ Post Reply | Private Reply | To 1 | View Replies]

To: LibFreeOrDie
Many science/medical journals these days prefer to give space to political harangues, rather than research.

There is 'research', but the 'research' is definitely evaluated more on its political implications than on its scientific merit.

13 posted on 02/15/2006 5:26:21 AM PST by Always Right
[ Post Reply | Private Reply | To 3 | View Replies]

To: Physicist
If peer review is too highly selective, that isn't a sign of being broken. It's a sign of health.

If the selection criteria is agenda based, then there is a problem. And today politics is a major driving force in what makes it to publication.

14 posted on 02/15/2006 5:28:05 AM PST by Always Right
[ Post Reply | Private Reply | To 10 | View Replies]

To: Always Right
And today politics is a major driving force in what makes it to publication.

What makes you say this?

15 posted on 02/15/2006 7:03:16 AM PST by Physicist
[ Post Reply | Private Reply | To 14 | View Replies]

To: AndrewC
To anyone familiar with funded research in the physical sciences, this problem was predictable decades ago.

The present day demands on an investigator to "Publish or Perish" are created not only from a desire to obtain job security via tenure, but a need to justify further research funding.

There was a time when patient, scholarly and well thought out experimental design resulted in the publishing of a handful of papers annually. While this body of work seldom involved results of a "breakthrough nature" in science, it was at least rigorously accurate and created a body of knowledge in a field that could be cited with confidence and often used by real geniuses in science to identify and support novel theories.

I contend that modern research in an academic setting no longer permits this patient scholarship and we have lost something as a result.
16 posted on 02/15/2006 7:50:16 AM PST by Panzerlied ("We shall never surrender!")
[ Post Reply | Private Reply | To 1 | View Replies]

To: Physicist
If peer review is too highly selective, that isn't a sign of being broken. It's a sign of health.

That certainly depends on the selection process. Anyway, I didn't choose the word "broken", the author did.

17 posted on 02/15/2006 6:47:37 PM PST by AndrewC (Darwinian logic -- It is just-so if it is just-so)
[ Post Reply | Private Reply | To 10 | View Replies]

To: AndrewC

bump


18 posted on 02/15/2006 6:51:47 PM PST by VOA
[ Post Reply | Private Reply | To 1 | View Replies]

To: Physicist
What makes you say this?

You don't think the politics of abortion, gay rights, and global warming don't have a major impact on what papers get published and which ones get rejected? Global Warming is probably the worst case.

19 posted on 02/15/2006 6:53:43 PM PST by Always Right
[ Post Reply | Private Reply | To 15 | View Replies]

To: AndrewC; gobucks; mikeus_maximus; MeanWestTexan; JudyB1938; isaiah55version11_0; bondserv; ...
(((Creationist Ping)))



You have been pinged because of your interest regarding matters of Creation vs. Evolution - from the Creationist perspective. Freep-mail me if you want on/off this list.

Colossians 1:16 "For by him all things were created: things in heaven and on earth, visible and invisible, whether thrones or powers or rulers or authorities; all things were created by him and for him."



And how much of a factor does personal bias and predjudice play in the dismissal of articles to peer-review? Evolutionists say that creationist articles are not peer-reviewed (by THEIR definition of "peers") and therefore are not scientific. What scientist who believes in evolution will review and approve an article which argues against his beliefs?

That being said, I happen to know that Answers in Genesis does rigorously review their articles, and each must pass inspection to be published.
20 posted on 02/16/2006 10:35:09 AM PST by DaveLoneRanger (I'm currently debating a big-time peace activist. I'll post it, so ping/mail me to read it when I do)
[ Post Reply | Private Reply | To 1 | View Replies]

To: AndrewC
Peer review has been broken since, at least, the early 90's.

It isn't about research anymore, or verifiable statistics, or case studies with large control groups.
It's all about the politics, baby.

21 posted on 02/16/2006 10:42:03 AM PST by Just another Joe (Warning: FReeping can be addictive and helpful to your mental health)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Just another Joe
Yeah. For top journals at least.
22 posted on 02/16/2006 10:50:02 AM PST by tallhappy (Juntos Podemos!)
[ Post Reply | Private Reply | To 21 | View Replies]

To: AndrewC

All we have to reasonably evaluate papers before publication is peer review. The author himself submits to peer review, in many cases, before submitting the paper to the journal by letting colleagues see the paper. Still, it is possible for an error to escape detection, and the error could be anything from a math error that few would be equipped to detect in the first place, to an error in reasoning due to a subtle fallacy that even fewer would be equipped to detect. It's not perfect even with the best of intentions, but what else can be done?


23 posted on 02/16/2006 10:55:59 AM PST by RightWhale (pas de lieu, Rhone que nous)
[ Post Reply | Private Reply | To 1 | View Replies]

To: AndrewC
Remember the randomly generated paper that was accepted by the "World Multi-Conference on Systemics, Cybernetics and Informatics 2005"?

The paper was generated ramdomly by three MIT students using SciGen.
24 posted on 02/16/2006 11:49:09 AM PST by Sopater (Creatio Ex Nihilo)
[ Post Reply | Private Reply | To 1 | View Replies]

To: DaveLoneRanger; Aetius; Alamo-Girl; AndrewC; Asphalt; Aussie Dasher; Baraonda; BereanBrain; ...

'Peer Review' is a misnomer. It's elitist control of the standards of debate, denying visibility to facts and ideas that are dangerous to the status quo.


25 posted on 02/16/2006 12:20:57 PM PST by editor-surveyor (Atheist and Fool are synonyms; Evolution is where fools hide from the sunrise)
[ Post Reply | Private Reply | To 20 | View Replies]

To: AndrewC

"It's something that's held "absolutely sacred" in a field where people rarely accept anything with "blind faith," says Richard Smith, former editor of the BMJ and now CEO of UnitedHealth Europe and board member of PLoS. "It's very unscientific, really."

Funny stuff!


26 posted on 02/16/2006 12:57:42 PM PST by mlc9852
[ Post Reply | Private Reply | To 1 | View Replies]

To: editor-surveyor
// 'Peer Review' is a misnomer. It's elitist control of the standards of debate, denying visibility to facts and ideas that are dangerous to the status quo.//

Yes, Peer review has become the antithesis of what its intent was.

Wolf
27 posted on 02/16/2006 1:10:58 PM PST by RunningWolf (Vet US Army Air Cav 1975)
[ Post Reply | Private Reply | To 25 | View Replies]

To: AndrewC

Peer review in the humanities was politicized long ago. There are journals I wouldn't send an article to on a bet, because they would reject it without bothering to look at it.

I'm afraid that there has been more and more politicization in the sciences, as well. That has always been the case to some degree, of course. Orthodox scientists with a stake in an accepted theory, for instance, aren't anxious to see it questioned or refuted. But nowadays the politics is much more widespread, because it involves leftist ideology and government grants supervised by ideologues. Foundation grants as well, I'm afraid.


28 posted on 02/16/2006 1:19:59 PM PST by Cicero (Marcus Tullius)
[ Post Reply | Private Reply | To 1 | View Replies]

To: AndrewC

Thank you for the ping.


29 posted on 02/16/2006 2:32:40 PM PST by Jo Nuvark ((Those who bless Israel will be blessed, those who curse Israel will be cursed. Gen 12:3))
[ Post Reply | Private Reply | To 1 | View Replies]

To: AndrewC

The problem with peer review is the problem with any system. It ultimately is political and is only as ethical as the people heading the heap. It is also subject to ideological motivations. If they don't like you, don't like what you're talking about or disagree on ideological grounds, you won't get reviewed. The article smells of something.. mostly handwringing.


30 posted on 02/16/2006 8:58:16 PM PST by Havoc (Evolutionists and Democrats: "We aren't getting our message out" (coincidence?))
[ Post Reply | Private Reply | To 1 | View Replies]

To: Havoc

These publications are political and cultural tools of the left. This is why I grow so tired of people on evolutionary threads stating "ID or creationist scientific research is not published in major journals"

I wonder why?

Howard Dean gets more respect as a physician than Behe ever does as a brilliant biochemist.


31 posted on 02/16/2006 9:44:03 PM PST by caffe
[ Post Reply | Private Reply | To 30 | View Replies]

To: caffe

Yep. And I wonder how many Republican ideas would be peer reviewed if Democrats were in charge of the review process.. Oops, I think we have a 40 year long example from which to draw on that point. And it largely came down on ideological lines.. hmm.. lol. But, evos are above such things.. (eye rolling) *snort*


32 posted on 02/16/2006 9:55:06 PM PST by Havoc (Evolutionists and Democrats: "We aren't getting our message out" (coincidence?))
[ Post Reply | Private Reply | To 31 | View Replies]

To: editor-surveyor

Thanks for the ping!


33 posted on 02/16/2006 10:07:36 PM PST by Alamo-Girl
[ Post Reply | Private Reply | To 25 | View Replies]

To: Havoc

yes, and to think this same strategy is not recognized by some conservatives when it comes to Darwin!


34 posted on 02/17/2006 1:57:41 PM PST by caffe
[ Post Reply | Private Reply | To 32 | View Replies]

To: Panzerlied

"I contend that modern research in an academic setting no longer permits this patient scholarship and we have lost something as a result."

I think you are right professor. Of course, new information is being accumulated at an every increasing speed. It is really difficult to stay on top of it. Seems that more scientists should stop trying to post in the "top" journals and settle for something less prestigous.


35 posted on 02/17/2006 8:53:41 PM PST by Sola Veritas (Trying to speak truth - not always with the best grammar or spelling)
[ Post Reply | Private Reply | To 16 | View Replies]

To: Sola Veritas

Indeed. Annual reviews have helped, but the sheer proliferation in journals, let alone articles can be intimidating. For all this, however, is the number of seminal articles proportionately greater?

The very forces driving the flood of publications are the ones which overwhelm us, and we spend too much time sifting dross in hopes of finding a few flecks of gold.

Maybe there should be a quota on publications. Like the WWII posters we should be secure enough in our professions to ask, "Is this paper necessary?"


36 posted on 02/18/2006 6:10:25 AM PST by Panzerlied ("We shall never surrender!")
[ Post Reply | Private Reply | To 35 | View Replies]

To: AndrewC
Experts also acknowledge that peer review is hardly ever to blame when fraud is published, since thoroughly checking data could take as much time as creating it in the first place.

Forgive me but then what good is it?

When my work is reviewed or I review the work of another the big thing we are looking for is error and/or fraud.

If peer reviewed means nothing more then "We found this idea interesting" then it becomes worthless as measure as to the value of the data presented.

37 posted on 02/18/2006 6:16:46 AM PST by Harmless Teddy Bear (Have you hugged your accountant today?)
[ Post Reply | Private Reply | To 2 | View Replies]

To: Panzerlied

"Maybe there should be a quota on publications. Like the WWII posters we should be secure enough in our professions to ask, 'Is this paper necessary?'"

I agree, but, of course, it will never happen. The young ones coming up are not "secure" enough, emotionally.


38 posted on 02/18/2006 10:42:55 AM PST by Sola Veritas (Trying to speak truth - not always with the best grammar or spelling)
[ Post Reply | Private Reply | To 36 | View Replies]

To: AndrewC
[ Is Peer Review Broken? ]

Depends on whos living in the hen house..
The chickens or the weasles.. When the clucking stops..
The guano has hit the fan..

39 posted on 02/18/2006 11:02:08 AM PST by hosepipe (CAUTION: This propaganda is laced with hyperbole..)
[ Post Reply | Private Reply | To 1 | View Replies]

To: hosepipe
When the clucking stops..

I guess the cluck stops here.

40 posted on 02/18/2006 11:14:32 AM PST by AndrewC (Darwinian logic -- It is just-so if it is just-so)
[ Post Reply | Private Reply | To 39 | View Replies]

To: AndrewC
[ I guess the cluck stops here. ]

One can hope... But with academia processing many brain washed socialist journalism majors as we speak.. not likely..

41 posted on 02/18/2006 11:29:00 AM PST by hosepipe (CAUTION: This propaganda is laced with hyperbole..)
[ Post Reply | Private Reply | To 40 | View Replies]

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.

Free Republic
Browse · Search
News/Activism
Topics · Post Article

FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson