By Guest Blogger Robert Dellavalle, MD, PhD, MSPH, Associate Professor of Dermatology and Public Health, University of Colorado and the Colorado School of Public Health
“To maintain our edge . . . we’ve got to protect our rigorous peer review system and ensure that we only fund proposals that promise the biggest bang for taxpayer dollars . . . that’s what’s going to maintain our standards of scientific excellence for years to come.” –President Barack Obama on the 150th Anniversary of the National Academy of Sciences, April 29, 2013
Peer review may be scientifically untested, expensive, unreliable, and biased [Guthrie et al. 2015; Ginther et al. 2011], but it’s the best we’ve got–so let’s not tweak it too much. That’s the gist of a recent New England Journal article examining current National Institutes of Health (NIH) peer review processes for reviewing grants [Lauer and Nakamura, 2015]. Support for the status quo defaults from the lack of acceptable proven alternatives. Alternative grant rewarding systems do exist: one example is awarding grants according to the past performance and creativity of applicants (as the MacArthur Awards do), and another example is creating milestone competitions to reach large end goals (as the Human Genome Project did). But the outcomes (number of publications, positive changes in clinical practice, etc.) of these alternatives have not been compared rigorously to the outcomes of selecting grants by peer review.
The NIH budget doubled 15 years ago. This expanded the number of science trainees, leading to more NIH grant applications from more people today. These applicants are also submitting more applications due to the increasingly competitive funding levels. This provides a dilemma for the NIH—with an ever-tightening budget, should it continue to fund high-risk projects that score highly with peer review or adapt other review systems that might better identify projects with greater scientific impact?
Do you ever think outside the box (Fig.1)? My personal NIH funding history provides a case report of a high-risk proposal. At the start of my academic career my wife Lisa Schilling, like all good internists, was reading the New England Journal of Medicine in bed late one night and pointed out to me that a large randomized trial of statins for the prevention of heart disease noted a lower rate of melanoma in those receiving treatment. Ah ha! I thought—perhaps this was the magic bullet to cure melanoma—or at least a new chemopreventive agent. Since many large, high quality statin clinical trials had already been completed, I proposed examining the effect of statins using meta-analysis of individual patient data from those trials. I submitted the project to seventeen funding agencies and was reminded repeatedly of Alain de Botton’s quote “Most business meetings involve one party elaborately suppressing a wish to shout at the other: ‘just give us the money’.” In the end the NIH National Cancer Institute was the source of funding for my grant that found that statins were not the magic bullet for preventing melanoma [Freeman et al. 2006].
Subsequently my research has turned to another unconventional topic–educating tattoo clients about the risk of UV radiation for ruining their tattoos and the canvas for their body art (their skin). This time my collaborators and I have not considered another funder besides the NIH. So, given my experiences, I strongly favor the NIH funding high risk, outside-the-box projects despite the messiness. But the question of how best to achieve this goal will remain until rigorous methods compare alternative grant reviewing processes [Azoulay 2012].
Azoulay P (2012) Turn the scientific method on ourselves. Nature 484:31–32.
Freeman SR, Drake AL, Heilig LF, Graber M, McNealy K, Schilling LM, and Dellavalle RP. (2006) Statins, fibrates, and melanoma risk: a systematic review and meta-analysis. J Natl Cancer Inst 98:1538-46. http://jnci.oxfordjournals.org/content/98/21/1538.long Accessed November 25, 2015.
Ginther DK, Schaffer WT, Schnell J, Masimore B, Liu F, Haak LL, and Kington R. (2011) Race, Ethnicity, and NIH Research Awards. Science 333:1015-1019.
Guthrie S, Guérin B, Wu H, Ismail S, and Wooding S. Alternatives to Peer Review in Research Project Funding 2013 Update. http://www.rand.org/pubs/research_reports/RR139.html Accessed November 25, 2015.
Lauer MS and Nakamura R. (2015) Reviewing Peer Review at the NIH. New Eng J Med 373:1893-4.