A Radical Revamp of Peer Review?

BETHESDA, MARYLAND--Scientists conducting a sweeping examination of the peer-review system at the National Institutes of Health (NIH) are considering some radical ideas to revamp the process, they revealed today. At a meeting here of the advisory committee to NIH Director Elias Zerhouni, members debated everything from doing away with the current scoring system on grant proposals to incentives that might improve the quality and motivation of reviewers.

Although peer review is still considered a cornerstone of science, it is experiencing new pressures. The average first-time NIH grantee is getting older, NIH budgets are nearly flat, and science has grown more complex. Furthermore, "we are desperately worried" about new investigators, who find it hard to land grants and may turn away from science, says Mary-Claire King, a geneticist at the University of Washington, Seattle, who was not a member of the peer-review team.

NIH last examined peer review 8 years ago in its so-called Boundaries report, but not all grants were covered. This time, "Elias said, 'Look at the whole thing,' " says Keith Yamamoto, executive vice dean of the University of California, San Francisco (UCSF), School of Medicine, and a biochemist there. In June, Zerhouni set up two working groups--one inside NIH and one outside--to solicit and synthesize comments from the scientific community (Science, 15 June, p. 1553).

There was no shortage of ideas; more than 2600 comments poured in. At today's meeting, Yamamoto, who co-chairs the external working group, and Lawrence Tabak, director of NIH's National Institute of Dental and Craniofacial Research and co-chair of the internal and external working groups, presented ways to reduce the administrative burden of grant writing and reviewing while funding the very best science. One possibility included an "editorial board" model, in which grant proposals that contain certain technical details could be sent to experts who would consider those elements alone and report back to the study section. This could help reduce the size of study sections, which have ballooned to as many as 80 people to accommodate all the expertise needed.

Other ideas include encouraging reviewers to be more direct about proposals that seem hopeless and seek fewer revisions; having each reviewer rank only the top 10 applications rather than scoring them all; and finding ways to get and keep the best reviewers in the system--including permitting reviewers to rate the reviews of their colleagues, which they can’t do now. Some proposed briefing reviewers on work NIH already funds to help avoid overlapping grants. "Are we studying the same protein head to toe?" Tabak asked.

For those applying for grants, possibilities being seriously considered include streamlining applications from 20 pages to seven, essentially doing away with preliminary data and instead focusing on a project's potential impact; and allowing resubmission of grant proposals only rarely, rather than the usual two times permitted today. There are "a lot of tensions that we're trying to navigate," said Mary Beckerle, director of the Huntsman Cancer Institute at the University of Utah in Salt Lake City, who is a member of one of the working groups.

Now, say Yamamoto and Tabak, the goal is to synthesize the ideas that have bubbled up in the two different working groups and present firm recommendations to Zerhouni by February.

thanks neverdem.

Ernest Lawrence, a pure experimentalist... said, "Don't you worry about it -- the theorists will find a way to make them all the same." -- Alvarez by Luis Alvarez (page 184)

I must reiterate my feeling that experimentalists always welcome the suggestions of the theorists. But the present situation is ridiculous... In my considered opinion the peer review system, in which proposals rather than proposers are reviewed, is the greatest disaster to be visited upon the scientific community in this century. No group of peers would have approved my building the 72-inch bubble chamber. Even Ernest Lawrence told me that he thought I was making a big mistake. He supported me because my track record was good. I believe U.S. science could recover from the stultifying effects of decades of misguided peer reviewing if we returned to the tried-and-true method of evaluating experimenters rather than experimental proposals. Many people will say that my ideas are elitist, and I certainly agree. The alternative is the egalitarianism that we now practice and that I've seen nearly kill basic science in the USSR and in the People's Republic of China. -- ibid (pp 200-201)

Also, publications as Nature or Science really, really don't want to have to publish anything that will make their reviewers of previously accepted publications look like dolts.

An example of this was a refutation we wrote of the core experiment of a paper published in Nature Neuroscience that claimed the existence of a transmembrane retention signal in the first transmembrane (TM) domain of the alpha subunit of the muscle acetylcholine receptor (AChR). The author claimed that this signal kept the protein in the endoplasmic reticulum (ER) until it could be assembled with the other subunits whereupon it became masked and the assembled receptor could then be released for further modification or trafficking to the plasma membrane.

He introduced a mutation consisting mostly of alanines into the first TM domain, expressed that subunit in the COS7 cell line, and then measured cell surface and total binding using I-125 bungarotoxin (Bgt). The native protein expressed alone has little surface expression but will bind Bgt while retained in the ER. He claimed that the introduction of the mutation resulted in an over 100-fold increase in cell surface binding, indicating it was no longer being retained, presumably by the putative retention signal. He introduced the alanine mutation to other proteins that are normally retained in the ER and claimed them to have been trafficked to the cell surface. He introduced the putative retention signal into Tac and EGFR, normally trafficked to the cell surface, and claimed them to have been retained in the ER.

Using the paper as a guide, I engineered the same alanine mutation into the TM domain of the mouse AChR alpha subunit. I expressed it alone and together with the other three wild-type subunits in both COS7 and tSA201 cell lines (the tSA201 can be transfected much more easily, as seen in much greater expression). I failed to see any increase in cell-surface Bgt binding with the mutation. I saw a reduced amount of intracellular binding. In addition, when expressed with the other three subunits, there was also a great decrease in the amount of Bgt binding. Western blots of mutated and wild-type subunits showed that the amount of protein expressed remained unchanged, only the binding capacity of the mutated form was reduced. I also expressed the putative retention signal in the transmembrane domain of alpha7/5HT3, a chimeric Bgt-binding protein that is robustly expressed on the cell surface at levels thousands of times higher than the alpha subunit alone as well as CD8 alpha, a type 1 single pass transmembrane protein. Contrary to his results, I saw no reduction in either surface expression or Bgt binding.

Because of these results, I obtained from the author samples of both his original alpha subunit as well as his mutated variety. I had both sequenced and then compared them to our alpha subunit DNA sequence. His wild-type sequence matched exactly, but his mutated variety had something not discussed in the Nature Neuroscience paper. Immediately upstream of the alanine mutations there were two single base pair deletions. Downstream, immediately after, there was a single base pair deletion. This meant that the DNA sequence for the alanines was out of frame. Instead of QRLAAAAIVAVI, the intervening sequence was actually QRLAAAAIVAVI. The downstream deletion frameshifted everything back and the full length alpha subunit was produced. At first I thought we had found the answer to the differences in our binding and expression and his published data. I then expressed both his and our wild-type and mutated DNA in both his pSM and in our pRBG4 vectors in both COS7 and tSA201 cell lines. Although the expression was lowered using the pSM vector, there was no difference in either expression or binding between his frameshifted version and our version that matched his published AAAAIVAV sequence--both were less than wild-type.

The reviewers' main points in turning down this correspondence for publication were: 1. the researcher had gone on to do other experiments with his altered alpha subunit and that we had failed to "address" those results. Our point was that whatever he did subsequently with that construct had validity only as far as his initial claims were true about expression and binding. If we showed no increase in either expression or binding, using his own DNA in his or other cell lines, the results of subsequent experiments, however he got them, had no meaning. 2. extraordinary claims require extraordinary proof. Ha ha ha. I mean, we used his DNA in his cell line and introduced the putative retention signal into similar sorts of surface-expressed proteins and got nothing that he claimed.

We figured that if his claims couldn't be supported by duplications of his own binding experiments, what kind of extraordinary proofs do we need? For that matter, the sorts of experiments we did went directly to addressing his claims: we used Bgt to measure binding; we used Western blots to show unaltered levels of protein expression; we used tagged proteins to show both surface and intracellular distribution of the proteins. There wasn't really anything else that could have been done. And there was no reason for us even to have to try to go on to posit anything about what controls retention of the alpha subunit until after assembly. He claimed he knew. Our experiments showed a vastly different story. But he was already published in Nature Neuroscience showing something that, if true, was a really cool thing. Publishing a refutation without anything more wasn't sexy enough. All it did was put his other results in the can and make the initial reviewers look like dorks.

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.