Review

Review Process

Reviewer Guidelines

CaAwardNet: Reviewer Portal

Previous Panel Members

Review Process

The review panel process, like the application and institutional approval process, is also conducted online via CaAwardNet.

The grant application review process employed by OICR is founded on the principles of internationally accepted scientific peer review.

Scientific review panel members are selected for their individual expertise and not as representatives of any particular organization. Prior to receipt of any grant application package, panel members review and complete a Reviewer Agreement regarding nondisclosure of confidential information and declaration of real or perceived conflicts of interest.

Panel members are provided with scientific summaries and applicants’ and co-applicants’ names to determine their level of expertise on applications, and ensure that applications are appropriately assigned.

All members of the scientific review panel read each application. Only members who have been assigned as primary, secondary or external reviewers are required to create written reports. These reports form the basis for discussion during the panel deliberations. During panel deliberations, other panel members comment on any aspect of the application, and on the strengths or weaknesses of the PI/team.

The actual scoring process during review meetings is also conducted on CaAwardNet: panel members enter and save their scores online.

Proposals submitted to the PMRF will be reviewed by one of two standing panels – the Translational Research Panel or the Clinical Trials and Companion Studies Panel.

The Translational Research Panel is composed of members with a broad range of expertise in the relevant scientific disciplines. The Panel will use external reviewers when required.

The Clinical Trials and Companion Studies Panel is composed of members with background in academia and industry, as well as investigators from major clinical trials groups.
Both panels will evaluate the scientific merit of the applications and provide written critique using the following criteria:

Significance: Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

Investigator(s): Do the PIs, collaborators, and other researchers have appropriate experience and training to conduct the project? If they are established researchers, have they demonstrated an ongoing record of accomplishments that have advanced their field(s)? If the project is collaborative or multi-PI/multi-site, do the investigators have complementary and integrated expertise; are the leadership approach, governance and organizational structure appropriate for the project?

Innovation: Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed?

Approach: Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project? Are potential problems, alternative strategies, and benchmarks for success presented? If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed?

Environment: Will the scientific environment in which the work will be done contribute to the probability of success? Are the institutional support, equipment and other physical resources available to the investigators adequate for the project proposed? Will the project benefit from unique features of the scientific environment, subject populations, or collaborative arrangements?

Additional Review Criteria: As applicable for the project proposed, reviewers will consider linkage of the proposal to Personalized Medicine Research and whether the project is answering one of the five key strategic questions in OICR’s Strategic Plan.

Biostatistical review:

Most pre-clinical, clinical and laboratory studies that bring new drugs, diagnostic tests, bio-markers or clinical tools from bench to bedside require biostatistical expertise at the planning, execution and analysis stages. Additionally, modern translational and personalized medicine research often requires high performance computing abilities, bioinformatics expertise and substantial data management support. A biostatistical evaluation of all translational and clinical trials/companion studies applications for the PMRF competition will be conducted by expert biostatisticians whose written critique/feedback will be provided to the applicants. These reviewers will score and rate applications in addition to the primary and secondary scientific reviewers, and the same scoring scale/weight as described below will be equally assigned to primary, secondary, biostatistical and panel reviews.

Biostatisticians will examine multiple aspects of each research proposal. It is expected that study aims and hypotheses will be clearly stated, and that the study design and analysis plans will be consistent with accomplishing the study aims and testing hypotheses in an unbiased and objective manner. For clinical studies, there should be a description of the intended study population, and the method of accruing or selecting study subjects from that study population should be explained. For laboratory studies, experimental samples and systems should be clearly described. Study endpoints should be explicitly defined. Testable statistical hypotheses or quantifiable metrics should be formulated to address the stated study aims. Statistical analysis methods appropriate for addressing those aims and consistent with the experimental design should be described.

A justification for the study sample size should be provided. Justification should be based on criteria such as adequate statistical power to detect a specific effect, estimate a parameter of interest with desired precision, or meet some other characteristic. Assumptions used to calculate the sample size (or power, precision, etc.) should be explained in enough detail to enable the biostatisticians to reproduce the calculation whenever possible.

Commercialization Review:

The grant applications will be reviewed by OICR’s Vice-President, Commercialization and Chief Commercial Officer, for:

1. Commercialization potential of the Background or Arising Intellectual Property (IP) including an assessment of freedom to operate (FTO) issues with respect to Background IP:
1. The existence of Background IP owned, or co-owned, by third parties; and
2. Do obligations to any third parties preclude the ability to implement the Project or practice Arising IP following completion of the Project?
2. Budget provisions for protection of IP.
3. Rights to Arising IP are, or will be, clearly and unambiguously established.
4. Potential to secure funding from other parties.

Lay Reviews

A Lay Reviewer is an individual who is not currently an academic researcher, but who has a demonstrated interest in health and science. The Lay Reviewer provides a mechanism for ensuring good communication to public stakeholders and transparency of the peer review process. He/she does not rate applications but provides verbal comments on the applications’ lay abstracts, specifically on the extent to which the intent and importance of the proposed research is well explained and in a language clear to members of the general public. Lay Reviewers are invited to provide feedback on the proceedings of the review panel, such as the quality, quantity and variety of science reviewed, the structure and objectivity of the discussions, and any other general comments.

Although Lay Reviewers neither score nor appraise the scientific quality of the application, applicants are encouraged to pay special attention to writing lay abstracts to facilitate the understanding of the proposals by the panel. Lay Reviewers are allowed to ask questions and seek clarifications on the scientific questions addressed in the proposals. Lay summaries of proposals should be written so as to be understood by persons without a scientific background.

Triaging Process

To increase quality, improve efficiency, provide flexibility and help prioritize the workload of the Panels, after the initial round of reviews (primary, secondary and biostatistical), the OICR internal scientific committee will sort the applications in order of ranking from highest to lowest, and where mean rating of the three reviewers (primary, secondary and biostatistical) is less than 3.0. The committee will also identify those applications that show disparity between the three reviewer scores in order to have a full panel review of those applications. It is highly recommended that applications that receive a mean score lower than 3 out of 5 are not discussed at the panel, provided there is no objection from other committee members. This process allows the reviewers to focus their discussion on the most meritorious applications.

For applications scoring lower than 3 out of 5, panel members will not vote and no budget discussions will occur. The final ratings will be calculated as the mean of the three scores (Primary, Secondary and bio-statistical). No Scientific Officer notes will be taken.

At the panel meeting, the panel members will be presented the list of all applications, highlighting the triaged applications for their information. The primary, secondary and bio-statistical review members of the triaged application will have an opportunity to bring forward the application for full review by the panel if he/she was too stringent in scoring or there was disparity in the initial review scores.

Feedback to the applicants of triaged application will report the fact that their application did not receive full panel review and that they should consider the feedback of the reviewers to improve their re-submission.

Not Fundable Applications:

Applications that receive an overall average score of less than 3 will be triaged at the start of the panel meeting and will not proceed to undergo a full panel review.

At the start of the panel meeting, panel members will be presented with the list of all applications; triaged applications will be highlighted for the reviewers’ information.

Applications with a score of less than 3 and a disparity between the primary and secondary reviewer scores will be reviewed at the beginning of the panel review. The primary and secondary review members of triaged applications will have an opportunity to bring forward the application to full review by the panel if either review member feels that he/she was too stringent in scoring, or if there is disparity in the primary and secondary review scores. Applicants will receive an initial review summary and will be informed that their application has not been successful in receiving full panel review.

Fundable Applications:

Applications with overall average scores of 3 and above will proceed to full assessment and ranking at the panel meeting. This process allows the reviewers to focus their discussion on the most meritorious applications.

The scientific officer of each panel will record the most significant points raised during the discussion of each application without participating in the formal rating exercise. The reviewers’ written comments, together with the scientific officer’s notes, are used to generate a constructive critique of the proposal that is provided to all applicants.

During the evaluation process, panel members rate all applications using a rating scale similar to other granting agencies, namely:

Applications from each panel are ranked according to their mean score. Since the two panels have dissimilar sets of criteria to assess applications and meet on separate occasions, the absolute scores are not comparable across panels. The scores are used only to achieve a relative ranking of grants on the panel.

Applications will be funded from the top-ranked down as far as the quality of the applications and the budget will allow.

Funding cut-off values for each panel are determined as follows:

After the panel meetings, OICR staff meet with the two panel chairs to obtain opinions on appropriate cut-offs for each panel. OICR staff then examine the cut-offs in light of the available budget to determine the number of grants that can be funded; this information is discussed with the OICR Board of Directors. Cut-off values are likely to be different between the two review panels.