A little bias in peer review scores can translate into big money

first_img Country * Afghanistan Aland Islands Albania Algeria Andorra Angola Anguilla Antarctica Antigua and Barbuda Argentina Armenia Aruba Australia Austria Azerbaijan Bahamas Bahrain Bangladesh Barbados Belarus Belgium Belize Benin Bermuda Bhutan Bolivia, Plurinational State of Bonaire, Sint Eustatius and Saba Bosnia and Herzegovina Botswana Bouvet Island Brazil British Indian Ocean Territory Brunei Darussalam Bulgaria Burkina Faso Burundi Cambodia Cameroon Canada Cape Verde Cayman Islands Central African Republic Chad Chile China Christmas Island Cocos (Keeling) Islands Colombia Comoros Congo Congo, the Democratic Republic of the Cook Islands Costa Rica Cote d’Ivoire Croatia Cuba Curaçao Cyprus Czech Republic Denmark Djibouti Dominica Dominican Republic Ecuador Egypt El Salvador Equatorial Guinea Eritrea Estonia Ethiopia Falkland Islands (Malvinas) Faroe Islands Fiji Finland France French Guiana French Polynesia French Southern Territories Gabon Gambia Georgia Germany Ghana Gibraltar Greece Greenland Grenada Guadeloupe Guatemala Guernsey Guinea Guinea-Bissau Guyana Haiti Heard Island and McDonald Islands Holy See (Vatican City State) Honduras Hungary Iceland India Indonesia Iran, Islamic Republic of Iraq Ireland Isle of Man Israel Italy Jamaica Japan Jersey Jordan Kazakhstan Kenya Kiribati Korea, Democratic People’s Republic of Korea, Republic of Kuwait Kyrgyzstan Lao People’s Democratic Republic Latvia Lebanon Lesotho Liberia Libyan Arab Jamahiriya Liechtenstein Lithuania Luxembourg Macao Macedonia, the former Yugoslav Republic of Madagascar Malawi Malaysia Maldives Mali Malta Martinique Mauritania Mauritius Mayotte Mexico Moldova, Republic of Monaco Mongolia Montenegro Montserrat Morocco Mozambique Myanmar Namibia Nauru Nepal Netherlands New Caledonia New Zealand Nicaragua Niger Nigeria Niue Norfolk Island Norway Oman Pakistan Palestine Panama Papua New Guinea Paraguay Peru Philippines Pitcairn Poland Portugal Qatar Reunion Romania Russian Federation Rwanda Saint Barthélemy Saint Helena, Ascension and Tristan da Cunha Saint Kitts and Nevis Saint Lucia Saint Martin (French part) Saint Pierre and Miquelon Saint Vincent and the Grenadines Samoa San Marino Sao Tome and Principe Saudi Arabia Senegal Serbia Seychelles Sierra Leone Singapore Sint Maarten (Dutch part) Slovakia Slovenia Solomon Islands Somalia South Africa South Georgia and the South Sandwich Islands South Sudan Spain Sri Lanka Sudan Suriname Svalbard and Jan Mayen Swaziland Sweden Switzerland Syrian Arab Republic Taiwan Tajikistan Tanzania, United Republic of Thailand Timor-Leste Togo Tokelau Tonga Trinidad and Tobago Tunisia Turkey Turkmenistan Turks and Caicos Islands Tuvalu Uganda Ukraine United Arab Emirates United Kingdom United States Uruguay Uzbekistan Vanuatu Venezuela, Bolivarian Republic of Vietnam Virgin Islands, British Wallis and Futuna Western Sahara Yemen Zambia Zimbabwe Email Late last month, many scientists submitted their first grant proposals of the year to the National Institutes of Health (NIH), the largest U.S. funder of biomedical research. Each will be graded by a panel of external reviewers—scientists who volunteer to rate the merit of the ideas presented by their peers. The scores peer reviewers award will play a big role in deciding which proposals NIH funds, and the process is extremely competitive: In 2014, the agency funded just 18.8% of the more than 27,000 proposals for a bread-and-butter R01 grant.In recent years, however, some observers have been questioning whether merit alone determines the outcome of such peer reviews, which many agencies around the world use to award research grants. Some studies have found that certain geographic or demographic groups, such as minorities or researchers from certain states, can fare unusually poorly in funding competitions, raising concerns that bias—conscious or unconscious—is skewing scores. Other experiments have raised questions about the role of randomness and subjectivity in scoring, showing that two groups of reviewers can give the same proposals very different scores, leading to different funding outcomes.Now, a new computer simulation explores just how sensitive the process might be to bias and randomness. Its answer: very. Small biases can have big consequences, concludes Eugene Day, a health care systems engineer at the Children’s Hospital of Philadelphia, in Research Policy. He found that bias that skews scores by just 3% can result in noticeable disparities in funding rates. Click to view the privacy policy. Required fields are indicated by an asterisk (*)center_img Sign up for our daily newsletter Get more great content like this delivered right to you! Country The “results are astonishing—funding is exceptionally sensitive to bias,” says ecologist Ruth Hufbauer of Colorado State University, Fort Collins. And although the simplified simulation may not explain exactly what’s happening in the real world, it does offer “a framework for quantifying the big-picture financial effects of institutionalized bias,” says geobiologist Hope Jahren of the University of Hawaii, Manoa. “It shows how systemic bias against any group translates into fewer dollars and cents to [a scientist] belonging to that group, irrespective of other factors.”To explore the role of bias in peer review, Day created a simplified mathematical model of the review process. Like anatomical models that allow medical students to practice costly surgical techniques without hurting anyone, Day’s model allowed him to conduct an inexpensive—and victimless—funding experiment. First, Day created two classes of imaginary grant applicants: a “preferred class” and a “non-preferred class.” Each class submitted a pool of 1000 grants to a funder; the two pools were statistically identical in quality.Next, three computer-generated reviewers scored each grant. (That’s the number of reviewers used in many real-world competitions.) In an ideal world, the scores would match the intrinsic quality of the grant. In the simulation, however, Day introduced some more realistic randomness: Not all reviewers scored the same grant the same way. And in what outsiders call a creative move, Day based the simulation’s randomness on the differences he’s seen in scores given to his own, real-world grants.“This was a very clever approach,” Hufbauer says, because few researchers publicly report how their grants were scored, obscuring such variation. So Day’s study “highlights that public data on variability in scores are not readily available, yet are a key piece of the puzzle of who is and who isn’t funded,” she says.In the final step, Day introduced bias. For the grants submitted by the “non-preferred” investigators, the three reviewers could reduce their scores by a little, a lot, or not at all. (No reduction represented unbiased scoring.)Overall, Day found that he could begin to detect bias in grant scores between the two groups when it amounted to just 1.9% of the scores given to the “non-preferred” grants. And when biased reviewers took more than 2.8% off the scores awarded to grants from the nonpreferred applicants, the bias began to affect funding decisions—even though that number is smaller than the difference in scores produced by randomness.In practical terms, the results meant that nonpreferred investigators had to submit higher quality grants to get money, whereas preferred investigators could get relatively lower quality grants funded. For instance, in a simulation that assumed that the top 10% of grants were funded, and that bias reduced the scores of nonpreferred applicants by an average of 3.7%, the preferred applicants got 118 funded grants, compared with just 82 from nonpreferred applicants. (Unbiased scoring would produce a roughly equal number of grants awarded in each group.)The simplified simulation has limitations. For example, many funding agencies don’t rely on scores alone to award all grants; they have program managers who review the scores and use them as one, albeit important, guide to picking winners. And Day writes that “it remains unknown how much bias there is in the real world, and how much that bias influences scores—and thus awards—of grants.”But the study does highlight “how little reviewer bias is necessary to result in noticeable outcome biases,” he says. And although “bias may be small in comparison to reviewer randomness,” he notes, it can still give an edge to preferred groups. If funding agencies recognize that fact and figure out how to address subtle bias, he adds, they could improve the overall quality of the grant applications they fund.“That such a minor bias could change funding outcomes is frankly shocking,” Hufbauer says. She adds that it should be a reminder to those who review grant and employment applications that even subtle bias can alter decisions.last_img

Posts Tagged with…

Write a Comment

Your email address will not be published. Required fields are marked *