Peer review - not perfect but still the best system that we have

In an interview with ERC Magazine, Ted Hewitt, President of Canada’s Social Sciences and Humanities Research Council, talks about his agency’s approach to evaluating grant proposals and working with review panels. He explains the challenges of peer review and ways to make the process better. 

SSHRC awards hundreds of grants every year. What are the guiding principles in evaluation of the research proposals you receive? 

For the most part we base it on the peer review and upon the assessment of excellence of the project. This entails several criteria. For example, our Insight Grants have three groups of weighted criteria of excellence: “Challenge”, i.e., the aim and importance of the endeavour (40%), “Feasibility” i.e., the plan to achieve excellence (20%) and “Capability” i.e., the expertise to succeed (40%). There's also increasingly in our adjudication processes, a component of equity, diversity, inclusion, or EDI, in light of the importance these factors play in research excellence. We'll ask the researchers how their teams are diverse or inclusive or how they can be made to be, and whether their research has implications for equity, diversity, and inclusion. And based on that EDI screen, we will reject applications that don't meet these criteria. All told, we select for research excellence.

And who sits on your review committees?

We want people from a variety of backgrounds. Most of them come from post-secondary institutions at various levels within the academy, but sometimes committee members will come from private or not for profit sectors. It's Canada, so we must make sure as much as possible, that we have peer reviewers who are able to review in English and in French because we're committed to receiving, adjudicating, and funding research in both official languages. We work exclusively with volunteer committees and we will compensate for travel or for expenses related to the actual review process itself.

we have peer reviewers who are able to review in English and in French

I mentioned earlier equity, diversity, inclusion. We apply these values also when we are looking for peer reviewers to make sure that we have representative panels with respect to women, Indigenous people, persons with disabilities and visible minorities. We also accept and adjudicate Indigenous research proposals. In our commitment to reconciliation, we have to recruit peer reviewers who are uniquely able to review these proposals. Finally, we have a regional focus as well: small institutions, large institutions… It becomes quite a complex matter, not only to find peer reviewers for subject matter review, but also to make sure they're properly representative of the country. 

How about getting researchers from abroad to assess your grant proposals? Your bilingual reviews may pose a challenge here.

It's true not every reviewer will have that capacity. During the review process, we need to make sure that the merit review is done appropriately, and that French-language and English-language proposals are reviewed by those who have the skills to do so. If it’s a committee for Insight Grants, we will send project proposals out for peer review to people who are fluent in the language of the proposal. Then the proposals go back to a panel of experts and we need to make sure that at that stage, the people in the panel are able to read and comment on the proposal itself within the language of submission. That is a challenge, and we agree on that.  

There are manuals and guidelines that are provided to peer reviewers and the panel of experts with a whole host of information around unconscious bias, on how to provide feedback on the need  to focus on the weighted criteria of excellence or managing conflicts of interest, confidentiality and so forth. And one thing we also do, which is quite interesting, is that we have observers in the peer review process. Their sole purpose is just to watch the process and then to report back on what went well and the things that didn't work so well. So, we're able to incorporate these things as improvements for next time.

we have observers in the peer review process

You launch a call for proposals, you receive applications and then what do you do with the submissions? Do you divide them into sub-panels for difference disciplines? How do you assign them to reviewers?

Let's use Insight Grants as an example, which is our biggest and most popular funding program. As people submit their projects, they'll select the subject areas from drop down menus. These are all reviewed by SSHRC staff and then our staff will determine how to aggregate proposals within disciplines and across topic areas or subject areas to try and ensure that they are reviewed by the appropriate committee. That takes a bit of skill, and it takes a bit of time, but it's absolutely critical. It gets very tricky, obviously, when proposals are multidisciplinary or cross disciplinary, but that's part of the job of the review and the filtering and the allocation of the proposals as they come in.

Do you have fixed committees or you create those committees depending on what proposals are submitted each time?

We used to have fixed committees a few years ago and then we moved now towards a more fluid model where we see what's actually in play. We can predict at some level from previous years what's likely to come in heavier or lighter in terms of areas.

Peer review is a predominant way of selecting proposals for funding. But there are also some critical voices saying it costs a lot, it takes too much time, there are biases and inconsistencies. Are you considering experimenting with some alternative approaches such as partial randomization or AI?

Yes, we have. We're running programs on behalf of the other agencies, with multidisciplinary applications. There we have tried some different things just given the nature of these competitions and their complexity. For example, when we're looking for peer reviewers, we tend to select reviewers who have experience with multidisciplinary research, as opposed to a mix of reviewers with deep disciplinary knowledge.  We are also using bio-sketches, as opposed to full CVs, whereby you're providing summaries of your research accomplishments to date. We have experimented with randomization and lotteries through this programme. We assess applications on the basis of merit within a category of acceptable/not acceptable, and then we funded the acceptable ones on the basis of a lottery. We also consider the Equity, Diversity, Inclusion screen to be critical for excellence insofar as it allows for the inclusion of a very broad array of researchers, research ideas and Indigenous knowledge systems. 

We also consider the Equity, Diversity, Inclusion screen to be critical for excellence insofar as it allows for the inclusion of a very broad array of researchers, research ideas and Indigenous knowledge systems.

We've been experimenting with all of these things and others - double-blind review is another methodology that we have used, and we're going to continue to do that through these very specialized multidisciplinary programs. And then, in future, we'll see how well they work, and we may start to incorporate some of these ideas into our mainstream peer review process.

So, in the end the peer review remain the core of the system.

I'd say peer review is not perfect but it's still the best system that we have. We as a community take on the responsibility to provide review, to seek out and fund meritorious proposals. I think, at the end of the day, we need to do everything we can to make it better by adhering to our principles and talking to each other.

I'd say peer review is not perfect but it's still the best system that we have. 

We just joined the Research on Research Initiative in the UK. We have some special challenges that I was mentioning around Indigenous research, making sure that we're funding projects that primarily are driven by and undertaken by and with Indigenous communities. We have linguistic challenges. But we do our best to provide the highest level of peer review that we're able to. 

Ted Hewitt was appointed president of SSHRC in March 2015. He is the current chair of the Canada Research Coordinating Committee and served as its inaugural chair from 2017 to 2019. Dr Hewitt was vice-president, research and international relations, at Western University in London, Ontario, from 2004 to 2011, where he had been a professor of sociology since 1989. He was also a public policy scholar at the Brazil Institute of the Woodrow Wilson International Center for Scholars in Washington, D.C. His current research focuses on national and international innovation systems, with emphasis on the roles of universities, industry and government in promoting economic prosperity in Latin America and beyond. He is co-chair of the Canada-Brazil Joint Committee for Cooperation on Science, Technology and Innovation, and a member of the board of the Brazil-Canada Chamber of Commerce. He holds a PhD in sociology from McMaster University. 

The Social Sciences and Humanities Research Council (SSHRC) is the federal research funding agency that promotes and supports research and research training in the humanities and social sciences. SSHRC also oversees the delivery of a number of tri-agency programs, including the Canada Research Chairs and other research chairs programs, and the New Frontiers in Research Fund, which supports international, interdisciplinary, fast-breaking and high-risk research.