An international expert panel on peer review convened by the Canadian Institutes of Health Research (CIHR) said in a recent report that the “intent and logic” of reforms that largely replaced face-to-face peer review with an electronic review process are “sound,” despite “implementation failure.”
But in a sign CIHR’s problems might be far from over, this conclusion has triggered more rebuke from critics of the reforms. (CIHR declined a request for an interview about the report.)
“The panelists are applauding the innovation at CIHR, despite its irresponsible consequences,” said Dr. Jim Woodgett, director of research at the Lunenfeld-Tanenbaum Research Institute in Toronto.
Last January, Woodgett warned the expert panel that Canadian researchers have been treated by CIHR like “guinea pigs in an uncontrolled experiment that has left many without funding.” His main criticism of the report is its assertion that CIHR should all but abandon face-to-face peer-review panels, except for a “limited number of ad hoc committees” at the final stage of review.
But even these are dispensable, wrote the international experts: “While this might be done electronically,” they explained, “the circumstances that have evolved recently are such that for a short period of time it may be appropriate to revert to face-to-face review for this stage to help rebuild confidence.”
The panel’s chairperson, Sir Peter Gluckman, serves as chief science advisor to the prime minister of New Zealand. He says the problems with face-to-face panels include cronyism, undeclared conflicts-of-interest, unresponsiveness towards innovative research, and high administrative costs and burdens.
“The literature also raises questions about groupthink and biases,” said Gluckman. “There is nonexistent or very marginal evidence for the validity of face-to-face panels.”
One of the dozen studies cited by the expert panel in specifically concluding that “peer review in its traditional face-to-face format is subject to biases,” is an unpublished manuscript by Robyn Tamblyn, scientific director of CIHR’s Institute of Health Services and Policy Research. Although the manuscript was shared with the panel last year, it cannot be released to CMAJ because it is being considered for publication, said Tamblyn.
“The evidence against face-to-face panels is lousy,” said Woodgett, regarding the cited papers. “The face-to-face panels may have problems, but like democracy, it’s the best system we have.”
Gluckman did acknowledge the paucity of research into the validity of face-to-face panels, noting that they continue to be relied upon for review of research proposals in many countries, including New Zealand.
According to Woodgett, rather than following the panel’s advice, CIHR should follow the lead of the US National Institutes of Health (NIH), which continues to use face-to-face panels. This idea is strongly endorsed by Dr. Michael Hendricks, who spoke before the panel last January on behalf of the Association of Canadian Early Career Health Researchers.
“The NIH stuck with face-to-face panels for a very good reason,” says Hendricks. “You can’t replace gatherings of scientists with an overwrought algorithm.”
Matthew Herder, director of the Health Law Institute, also defended face-to-face peer review before the panel in January, on behalf of the Federation of Humanities and Social Sciences.
“All of the literature I reviewed before presenting to the panel indicates that, to assess interdisciplinary research proposals, you need to have people from all of the disciplines in the same room,” said Herder. “I saw no empirical evidence suggesting that virtual reviews work just as well as face-to-face panels.”