Driven by UHN's mission for research excellence,
Dr. Anna Gagliardi, Senior Scientist at Toronto General Hospital Research Institute (TGHRI), led an advisory group to redefine research assessment practices at UHN.
In January 2020, UHN signed the
San Francisco Declaration on Research Assessment (DORA). The agreement outlines principles that promote better approaches to assessing research impact than the traditional overreliance on journal and publication metrics.
"This commitment marked an important milestone," says Dr. Brad Wouters, UHN Executive Vice President of Science and Research. "We are changing research assessment in ways that provide a more complete understanding of our researchers' work in order to drive continuous improvement in research quality and impact.
"This is done as part of our larger strategy to advance our vision of
A Healthier World."
The DORA Advisory Group, which included researchers from across UHN, was created to put the principles outlined in DORA into action.
Two recent publications outline their approach and progress – one published in
PLoS One and the other in
Health Research Policy and Systems – and are the first publications featuring researchers from across all six UHN research institutes.
"Initially, we focused on identifying and prioritizing measures for assessing research performance according to the DORA principles," says Dr. Gagliardi. "These measures would help with hiring, annual review, promotion and tenure decisions."
Through a comprehensive review of literature, the group compiled an initial list of 50 ways of assessing research activity and grouped them into eight domains. These domains included but were not limited to research productivity, strategic value, funding, publications, innovations and impact.
"To consolidate and prioritize these measures, we conducted two rounds of surveys with a panel of diverse researchers," says Himani Boury, a previous practicum student at the Toronto General Hospital Research Institute and first author on the
Health Research Policy and Systems publication.
The findings led to the identification of the following ten priority research assessment measures:
| High Importance | Moderate Importance |
|---|
| Advancing existing knowledge | Challenges to research productivity |
| Innovative research plans | Potential to improve health or health care |
| Independent funding support | Team science |
| Relevant research outputs | Collaboration |
| Publication content quality | Recognition by professional/academic groups |
"Our next focus was exploring the potential barriers to implementing and adopting these measures on an organizational level," says Himani. "To do so, we conducted 18 interviews with researchers and research support staff from across UHN's six research institutes.
"This shed light on a several strengths and barriers to implementation."
Participants found the measures to be comprehensive, relevant across disciplines and rigorously developed. The reporting template for the measures was also deemed easy to understand and use. However, concerns were expressed over the measures' relevancy across research disciplines. Another concern raised was that the complexity of the measures would make them more time-consuming to use.
To overcome these barriers, various strategies were suggested, including high-level endorsement of the measures, an official launch accompanied by a multi-pronged communication strategy, training for both researchers and evaluators, administrative support or automated reporting tools, guidance for evaluators, and sharing of approaches among research institutes.
"Addressing barriers and implementing supportive strategies is crucial for the adoption of prioritized measures. These findings urge research organizations worldwide to reassess evaluation practices, embrace DORA-compliant measures, and work collaboratively to foster a transformative shift towards prioritizing quality and impact in research assessment," concludes Dr. Gagliardi.
The work of the DORA Advisory Group will be continued by the Research Assessment Implementation Committee led by Drs.
Michael Hoffman and Gagliardi.
This work was supported by UHN Foundation and The Princess Margaret Cancer Foundation.
Dr. Mathieu Albert is an Emerita/Emeritus Scientist at The Institute of Education Research (TIER) at UHN and an associate professor in the Department of Psychiatry and the Institute of Health Policy, Management and Evaluation at the University of Toronto (U of T). Dr. James CL. Chow is a clinician scientist at Princess Margaret Cancer Centre and an associate professor in the Department of Radiation Oncology at U of T. Dr. Ralph S. DaCosta is a Senior Scientist at Princess Margaret Cancer Centre and holds a Cancer Care Ontario Research Chair in Cancer Imaging. Dr. Anna Gagliardi is a Senior Scientist at the Toronto General Hospital Research Institute and a professor in the Department of Surgery, the Institute of Health Policy, Management and Evaluation and the Institute of Medical Science at U of T. Dr. Michael Hoffman is a Senior Scientist at Princess Margaret Cancer Centre and an associate professor in the Departments of Medical Biophysics and Computer Science at U of T. Dr. Behrang Keshavarz is a Senior Scientist at UHN's KITE Research Institute and an adjunct assistant professor at the Department of Psychology at the Toronto Metropolitan University. Dr. Pia Kontos is a Senior Scientist at UHN's KITE Research Institute and a Professor at the Dalla Lana School of Public Health at U of T. Dr. Jenny Liu was a postdoctoral research fellow at UHN's Krembil Research Institute. Dr. Mary Pat McAndrews is a Senior Scientist at UHN's Krembil Research Institute and a professor in the Department of Psychology at U of T. Dr. Stephanie Protze is a scientist at UHN's McEwen Stem Cell Institute and an assistant professor in the Department of Molecular Genetics at U of T.
Administrative support for the DORA Advisory Group was provided by the Research Analytics Department of UHN's Research Solutions and Services.
