Inside a peer-reviewed grant competition

When writing about cancer research, your correspondent finds himself using the same phrase over and over. Whenever a researcher receives a grant or award from the Ontario Institute for Cancer Research (OICR), the article to announce it always states that she or he was awarded the funds in a “competitive peer-reviewed grant competition.”

In the back of my mind, I've always known what this phrase means: before investigators receive funding, they have to prove to people who are knowledgeable about their field that the proposed research is scientifically sound, within the mandate of the granting agency, and has a realistic chance of success.

But what does a peer-reviewed grant competition actually look like? Who are these peers, and where do they come from? Do the scientists actually compete with one another? How?

Recently, I received an invitation to go find out for myself. Dr. Bob Phillips, Deputy Director of OICR, and Dr. Teresa Petrocelli, Director of Grants and Awards, invited me to sit in on a grant competition to see what the process looks like.

Before leaving the office, I did a bit of research and was somewhat disappointed to learn there wouldn't be any scientists facing off against one another in a battle for funds. In fact, the people who submit applications are not even invited to present to the panel that decides whether they will be funded. Instead, they have to prepare written applications in a standardized format and submit them to OICR months before the competition.

The application is forwarded to each of the panel members and the chairs. Each panel has two chairs, meaning that each chair has to review half of the applications before coming to the meeting. Each application is also assigned to two reviewers, who prepare a report that they deliver orally when they come to the meeting. Each application is also assigned to a reader, who is expected to read the application closely and contribute to the discussion of the application they have been assigned. The night before the meeting, OICR makes available the anonymous reports to each of the panel members, allowing them to see the reviewers’ feedback in advance of the meeting, which expedites the discussion the following day.

Beyond these basic details, your correspondent didn't really know what to expect upon arrival at Toronto's Delta Chelsea hotel early on a cold Thursday morning in early February. After I got somewhat lost in the maze of corridors around the Delta's convention centre, a helpful staff member directed me to a small meeting room. When I stepped in, the session was about to begin. Petrocelli and her staff had set aside a few chairs at the back of the room for observers – me, and two new members of her team who would be sitting in for the morning to learn how the awards their group administers are granted. Petrocelli began the day by welcoming the panellists and giving them an overview of OICR.

It seemed a bit strange that a panel of scientists who are knowledgeable about cancer research would need an overview of OICR, until I started looking around the table and noticing people's name cards. Aside from Phillips and Dr. Alex MacKenzie, an Ottawa-based clinician-researcher who was serving as panel co-chair, none of them were from Ontario. This also seemed strange. Shouldn't the people who decide how OICR's Cancer Research Fund is distributed be familiar with research in Ontario?

It turns out the opposite is true. “The Cancer Research Fund (CRF) was set up as a way to fund the best science,” explains Phillips. “Who the researchers are, or which institution they are affiliated with, is not important when we're talking about which projects have the most scientific merit. We make decisions based on the science that the researchers are proposing. The projects also have to be feasible – but again, to determine if a scientific project is feasible requires scientific expertise and knowledge of research administration, which doesn’t have to be specific to Ontario.”

Recruiting panellists from other provinces and the United States has an added advantage. As with any grant review process, the OICR CRF competitions require panellists to step out of the room if they are affiliated with or close colleagues of researchers who submit applications. This could pose a problem if all the panellists were active researchers in Ontario, since the province's cancer research community is closely knit and panellists would likely have to declare conflict of interest on many projects in their areas of expertise.

It was also surprising to learn that two of the panel members were not scientists. Lay panellists are familiar with cancer issues, but they are not expected to have any background in science. Because they do not have the knowledge needed to assess the projects' merit, they do not have a vote. Instead, their role is to act as representatives of the people of Ontario and provide non-scientific opinions for the panellists who do not live here, and also to provide feedback to the people who submit projects. This feedback tells them if the project sounded worthwhile or promising from a layperson's perspective – which can be valuable for researchers who want to ensure their priorities are aligned with those of the general public.

The voting members of the panel are people who do have a background in science. On the day I attended, several members were academic researchers; one was a chemist from industry; one was director of a cancer institute in New Jersey. Their task was to review projects related to translational research. Earlier in the week, Petrocelli and her staff hosted a separate panel of clinician-researchers that reviewed projects related to clinical trials. “OICR funds research in a wide variety of areas, and many of the applications are highly technical, so it's important for us to have the right experts in the room to discuss the right projects,” she explains.

Before the panel began its deliberations, Petrocelli explained how the scoring of applications would work.

Each application is assigned to two reviewers, who read it, score it on a five-point scale and bring their comments to the group. The project chair first determines if the project should be discussed by checking that both primary and secondary reviewer preliminary scores are above 3.0. Because the CRF receives far more applications than it is able to fund, it uses this “triage” stage to cut projects that are not competitive.

If the score is above 3.0, the project chair invites the primary reviewer followed by the secondary reviewer, to deliver oral reports. To expedite the process, the second reviewer typically tailors her or his comments to areas that the first reviewer has not discussed, or to areas where she or he disagrees with the first reviewer. After the presentations, the chair invites the panel to discuss the project.

Next comes the voting stage. The chair first asks if the primary and secondary reviewers would like to reconsider their preliminary scores in light of their discussion. Often, the result of the discussion is that one or both of the reviewers will bring their scores closer together. These scores are then averaged to create a “consensus score” that is used for reference by the other panellists. After the consensus score is reached, panellists vote by secret ballot. If their individual score deviates from the consensus score by more than 0.5 points in either direction (i.e., 10 per cent of the scale) they are asked to inform the chair that they will be scoring outside the range of the panel. Normally, scores are close to the consensus score, but it’s possible for the panel to be split on some applications. The average score of the full panel is the final score used to rank which applications will be funded.

With these details out of the way, the chair announced it was time to talk science. And for the next eight hours, that's exactly what the panel did.

Your correspondent isn’t a scientist and won’t pretend that he understood every word as the scientists debated applications in what seemed like an incredible amount of detail. But watching how the panellists interact and come to decisions was fascinating.

Several times, the medicinal chemist jumped into the discussion to say an application seemed to have merit at the basic research stage, but expressed doubts that it could be turned into a drug because of experiences he’d had with similar compounds. This level of detail is important. The awards being decided amount to hundreds of thousands of dollars, and OICR reviews more than five times as many applications as it is able to fund, so the content of the applications must be reviewed carefully to ensure the researchers’ plans are feasible and realistic.

It’s often been said that research is an art as well as a science. Seeing a room filled with top talent split along scientific lines, your correspondent would be tempted to say that despite the highly scientific process of evaluating research grants, deciding who gets funded is an art as well. At a few points, the panellists found themselves on different sides of the same scientific issue, with half the room arguing that an application should be funded because it was scientifically ambitious, and the other half arguing that the applicant was relying on assumptions that hadn’t yet been proven.

Needless to say, there is a lot of potential for these discussions to run off course. The role of the chair is to keep the panellists back on track, guiding them toward a consensus – or in some cases where the issues could not be resolved, toward a split vote.

There is one more step before moving on to the next application: the Scientific Officer, a person with a background in science who sits at the table and takes notes, reads notes that were taken during the discussion back to the panel for comment and clarification. This is especially important for the applications that are not accepted; the researchers who made the applications are allowed to see this feedback, which could help them improve their applications for future submissions. Even the applications that did not receive a score of 3.0, the minimum required for review by the full panel, get feedback from the two reviewers.

It was quite impressed to see how well the Scientific Officer at the panel I attended, Dr. Isabella Tai, was able to take notes on the discussions and read a synopsis back to the group without any time to review. For the vast majority of projects, the panellists nodded their heads, agreeing that she’d accurately captured not just the spirit, but also the nuances, of their comments.

The few times that panellists did ask Tai to make revision, it was almost always for the same reason: the reviewers had been frank about the shortcomings or limitations of the application, but rather than give this negative feedback to the researcher who submitted it, they felt it would be more productive to provide constructive criticism aimed at how they might increase the chances of getting their application funded – or, to use the language of grants and awards, to make it more “fundable.”

On other occasions, panellists seemed to find themselves editing applications at the table.

“Why wouldn't they have taken it a step further, and also propose...” was a common refrain. So was, “If they'd only been more realistic with their plans!”

After a few of these comments, however, the chair would always cut off the discussion with a quote, which MacKenzie said he’d gleaned from Phillips at a panel that Phillips had once chaired.

“We can only review the application in front of us, not the one we wish they’d written.” With that, he'd check to make sure the Scientific Officer had recorded feedback for the benefit of the investigator, then move on to the next application. After all, there's only so much time – and many more applications to review.

Less paper, less 'committee drift'

At previous panels, OICR printed applications and mailed them to panellists before the meeting and used a paper ballot system at the meeting to score the applications. Starting this year, however, OICR’s Grants and Awards Department is using technology that almost completely removes paper from the process. Using a comibation of two tools, CaAwardNet and eReview. Researchers now forward applications electronically before the meeting. At the meeting, panellists use laptops to enter their scores.

“eReview saves us time and lets us get details about projects quickly. It’s amazing how the paradigm can shift away from paper.” – Lay reviewer Norman Grey

“eReview has tremendous promise for keeping us on track. When the discussion drifts, the panels waste time, and the applicants suffer because we don’t have as much time to provide them with feedback as we would if we stayed on track. eReview is a great check and balance against committee drift.”
– Panel Chair Dr. Alex MacKenzie