Driven by UHN’s mission for research excellence, Dr. Anna Gagliardi, Senior Scientist at Toronto General Hospital Research Institute, led an advisory group to redefine research assessment practices at UHN.
In January 2020, UHN signed the San Francisco Declaration on Research Assessment (DORA). The agreement outlines principles that promote better approaches to assessing research impact than the traditional overreliance on journal and publication metrics.
“This commitment marked an important milestone,” says Dr. Brad Wouters, UHN Executive Vice President of Science and Research. “We are changing research assessment in ways that provide a more complete understanding of our researchers’ work in order to drive continuous improvement in research quality and impact. This is done as part of our larger strategy to advance our vision of A Healthier World.”
The DORA Advisory Group, which included researchers from across UHN, was created to put the principles outlined in DORA into action. Two recent publications outline their approach and progress—one published in PLoS One and the other in Health Research Policy and Systems—and are the first publications featuring researchers from across all six UHN research institutes.
“Initially, we focused on identifying and prioritizing measures for assessing research performance according to the DORA principles,” says Dr. Gagliardi. “These measures would help with hiring, annual review, promotion and tenure decisions.”
Through a comprehensive review of literature, the group compiled an initial list of 50 ways of assessing research activity and grouped them into eight domains. These domains included but were not limited to research productivity, strategic value, funding, publications, innovations and impact.
(L-R) Himani Boury, previous practicum student at the Toronto General Hospital Research Institute; Dr. Anna Gagliardi, Senior Scientist at the Toronto General Hospital Research Institute.
“To consolidate and prioritize these measures, we conducted two rounds of surveys with a panel of diverse researchers,” says Himani Boury, a previous practicum student at the Toronto General Hospital Research Institute and first author on the Health Research Policy and Systems publication.
The findings led to the identification of the following ten priority research assessment measures:
| High Importance | Moderate Importance |
|---|---|
| Advancing existing knowledge | Challenges to research productivity |
| Innovative research plans | Potential to improve health or health care |
| Independent funding support | Team science |
| Relevant research outputs | Collaboration |
| Publication content quality | Recognition by professional/academic groups |
“Our next focus was exploring the potential barriers to implementing and adopting these measures on an organizational level,” says Himani Boury. “To do so, we conducted 18 interviews with researchers and research support staff from across UHN’s six research institutes. This shed light on a several strengths and barriers to implementation.”
Participants found the measures to be comprehensive, relevant across disciplines and rigorously developed. The reporting template for the measures was also deemed easy to understand and use. However, concerns were expressed over the measures' relevancy across research disciplines. Another concern raised was that the complexity of the measures would make them more time-consuming to use.
To overcome these barriers, various strategies were suggested, including high-level endorsement of the measures, an official launch accompanied by a multi-pronged communication strategy, training for both researchers and evaluators, administrative support or automated reporting tools, guidance for evaluators, and sharing of approaches among research institutes.
"Addressing barriers and implementing supportive strategies is crucial for the adoption of prioritized measures. These findings urge research organizations worldwide to reassess evaluation practices, embrace DORA-compliant measures, and work collaboratively to foster a transformative shift towards prioritizing quality and impact in research assessment," concludes Dr. Gagliardi.
The work of the DORA Advisory Group will be continued by the Research Assessment Implementation Committee led by Drs. Michael Hoffman and Gagliardi.
This work was supported by UHN Foundation and The Princess Margaret Cancer Foundation.
The members of the DORA Advisory Group included Drs. Mathieu Albert, James CL. Chow, Ralph S. DaCosta, Anna Gagliardi, Michael Hoffman, Behrang Keshavarz, Pia Kontos, Jenny Liu, Mary Pat McAndrews and Stephanie Protze. Dr. Mathieu Albert is an Emerita/Emeritus Scientist at The Institute of Education Research and an Associate Professor in the Department of Psychiatry and the Institute of Health Policy, Management and Evaluation at the University of Toronto. Dr. James CL. Chow is a Clinician Scientist at the Princess Margaret Cancer Centre and an Associate Professor in the Department of Radiation Oncology at the University of Toronto. Dr. Ralph S. DaCosta is a Senior Scientist at the Princess Margaret Cancer Centre and holds a Cancer Care Ontario Research Chair in Cancer Imaging. Dr. Anna Gagliardi is a Senior Scientist at the Toronto General Hospital Research Institute and a Professor in the Department of Surgery, the Institute of Health Policy, Management and Evaluation and the Institute of Medical Science at the University of Toronto. Dr. Michael Hoffman is a Senior Scientist at the Princess Margaret Cancer Centre and an Associate Professor in the Departments of Medical Biophysics and Computer Science at the University of Toronto. Dr. Behrang Keshavarz is a Senior Scientist at the KITE Research Institute and an adjunct Assistant Professor at the Department of Psychology at the Toronto Metropolitan University. Dr. Pia Kontos is a Senior Scientist at the KITE Research Institute and a Professor at the Dalla Lana School of Public Health at the University of Toronto. Dr. Jenny Liu was a Postdoctoral Research Fellow at the Krembil Research Institute. Dr. Mary Pat McAndrews is a Senior Scientist at the Krembil Research Institute and a Professor in the Department of Psychology at the University of Toronto. Dr. Stephanie Protze is a Scientist at the McEwen Stem Cell Institute and an Assistant Professor in the Department of Molecular Genetics at the University of Toronto. Administrative support for the DORA Advisory Group was provided by the Research Analytics department of UHN’s Research Solutions and Services.
Gagliardi AR, Chen RHC, Boury H, Albert M, Chow J, DaCosta RS, Hoffman M, Keshavarz B, Kontos P, Liu J, McAndrews MP, Protze S. DORA-compliant measures of research quality and impact to assess the performance of researchers in biomedical institutions: Review of published research, international best practice and Delphi survey. PLoS One. 2023 May 12;18(5):e0270616. doi: 10.1371/journal.pone.0270616.
Boury H, Albert M, Chen RHC, Chow JCL, DaCosta R, Hoffman MM, Keshavarz B, Kontos P, McAndrews MP, Protze S, Gagliardi AR. Exploring the merits of research performance measures that comply with the San Francisco Declaration on Research Assessment and strategies to overcome barriers of adoption: qualitative interviews with administrators and researchers. Health Res Policy Syst. 2023 Jun 5;21(1):43. doi: 10.1186/s12961-023-01001-w.
Currently over 23,000 individuals and organizations in 162 countries (coloured in yellow) across the world have signed the San Francisco Declaration on Research Assessment. (Source: sfdora.org)