Statistics and Research Methodology Training Needs in Medical Imaging
Abstract
Purpose:
The purpose of this study is to recommend improvements for research methodology and statistics training resources specifically designed for medical imaging residents, fellows, and faculty.
Methods:
In 2017, survey and interview questions related to existing standards and ideal characteristics for a statistics and research methodology training program in medical imaging were created and administered to a large academic department prospectively. To explore common usages of statistics and research methodology skills in medical imaging, articles published in 2013 from all departmental faculty members were retrieved from Scopus and a review was conducted retrospectively. A mixed methods approach was used, including standard protocols for online surveys, phone interviews, and a cross-sectional review.
Results:
All interviewees stated that a statistics and research methodology training program would be beneficial to the department. 51% of survey respondents either only had informal or no training experiences in statistics and research methodology. 322 articles were published in 2013 by 177 faculty members. The median journal impact factor was 2.7 (IQR = 1.7 – 3.7) and median h-index of the corresponding authors was 10 (IQR = 4 – 22). 65% of the articles considered in the review used statistical methods with 79% of these employing a “basic” level of complexity (basic I, basic II, intermediate, and advanced).
Conclusion:
Results from this study support the need to offer broad training in research methodology. A better emphasis on basic level statistics that enables a more efficient use of departmental resources may be connected to a more cost effective and enabling education program.
Introduction
Medical trainees and clinicians are responsible for making informed choices about patient management based on accurate interpretations of research evidence. This requires an understanding of statistics and research methodology [1]. This has been demonstrated in trainee and clinician populations, including emergency medicine and undergraduate medical school programs [2,3]. Clinicians also undertake impactful research, requiring statistical knowledge for analyses, and interpretation of results[4].
Postgraduate medical students find statistics difficult, are frequently uncertain about the meaning of statistical terms, and struggle with selecting research methodologies in their research [5]. Graduate medical students who have taken quantitative university courses, including mathematics, also perceived statistics with feelings of anxiety [6]. The needs and preferences of medical imaging trainees and clinicians are important when considering ways to support this group of learners in conducting high quality research and providing high standards of patient care. The purpose of this study is to recommend improvements for research methodology and statistics training resources, specifically designed for medical imaging residents, fellows, and faculty. For this study, research methodology and statistics training was analyzed in the context of skills to analyze results. The objectives were: (1) survey and interview medical imaging residents, fellows, and faculty to explore the optimal depth, learning environment, and set of content priorities for a statistics and research methodology program; (2) review articles published in one year from faculty members in the Department of Medical Imaging at the University of Toronto, in order to determine commonly utilized statistical methods and; and (3) use the results to assist in the design and implementation of a data science support unit within the department.
Methods
Study Design
A prospective mixed methods approach was used, including standard protocols for online surveys, phone interviews, and a cross-sectional review [7,8]. Verbal informed consent was obtained from interviewees, and The University of Toronto’s Health Sciences Research Ethics Board approved this study (#34191).
Online Survey and Phone Interview
Participants. Residents, fellows, and faculty from the Department of Medical Imaging were invited to participate in an online survey and a phone interview. The list of participants, including their contact information, was provided by Department of Medical Imaging. There were no exclusion criteria.
Outcomes. The standards for existing statistics and research methodology training opportunities were explored. Additionally, insights about the optimal depth, learning environment, and content for an ideal statistics and research methodology training program in a medical imaging department were to be identified through the survey and interviews.
Intervention. In 2017, quantitative data was collected online using Survey Monkey surveys (see Figures S1-S3). After surveys were completed, qualitative insights were collected through 10-minute semi-structured phone interviews. Questions 1-3 collected background participant information, including roles previously or currently held, formal research methodology or statistics course experiences, and preferred software. Questions 4-5 focused on gathering insights on content priorities, gained through comparing competent content areas in question 4 and the top three content areas for careers in question 5. Question 6 focused on ideal pedagogical formats, asking participants for the formats of learning they prefer. The phone interview elaborated on participants’ responses to the survey, following the same topics. Participants were given the option to participate in one or both parts of the study.
Data collection. Recruitment for the online survey was based on emails sent directly to residents, fellows, and faculty in the Department of Medical Imaging. Instructions, a link to the survey, and an invitation for a phone interview were included. Survey Monkey software was used, because it is reliable, well-known, and allows personalized question formats to be created. Participants were given the option to complete the surveys on any electronic device with internet access. Consent was obtained at the beginning of online surveys via a mandatory question that required answering before proceeding. To advance questions, user pressed a ‘Next’ button.
Phone interviews were conducted after the surveys and at a time that was convenient for the participant, scheduled via email. Responses were recorded using an audio recorder, if consent was provided, and recordings were later transcribed for qualitative analysis. No personal information was collected during the interview, and any personal information mentioned accidently was removed while transcribing. The phone interview was semi-structured, based on the order of survey questions. At the end, participants were asked to discuss any additional information they felt was valuable.
Data analysis. Quantitative data from the surveys were used to calculate univariate statistics (frequencies/percentages). Qualitative results were analyzed using inductive thematic analysis to identify any themes emerging from phone interviews with participants [9]. A question-focused content analysis approach was used to determine patterns of responses. The interview data was transcribed verbatim and anonymized from recordings, and was then coded and analyzed for key themes.
Review of Published Articles
Search Strategy. The articles published by the University of Toronto’s Department of Medical Imaging in 2013 were retrospectively searched on Scopus. Publications were retrieved by searching for faculty member names from the department within the list of authors.
Study Selection. Out of the 322 abstracts retrieved, a randomized sample of 170 articles (53%) were reviewed. A randomized sample was selected because this was an unfunded undergraduate project. A full review was not possible due to restricted resources and time. Simple randomization was used. No exclusion criteria were applied to full text articles, as all articles in the sample would be examined to find distribution of type of study, impact factor, h-index, and if applicable, level of statistical analyses. A flow diagram of the literature search is represented in Figure S4.
Outcomes. The review identified the difficulty and impact of the statistical methods that were most commonly used in articles published by the University of Toronto’s Medical Imaging Department.
Intervention. A review of abstracts from articles published from faculty in the University of Toronto’s Medical Imaging Department in 2013 was conducted.
Data collection. The full-text versions of the articles were independently evaluated by authors HL and AM. Data collected consisted of article citation, bibliometric indicators, study methodology and presence of statistical analysis. If statistical analysis was performed, the statistical software and methods were recorded. Disagreements in the data extracted by the two authors HL and AM were resolved by PNT.
Data analysis. For each abstract for studies that included statistical analysis, the statistical methods performed were categorized into four levels, based on standards in biostatistics consulting by our department: Basic I, Basic II, Intermediate, and Advanced (see Figure S5). Journal impact factors were retrieved from InCites Journal Citation Reports (Thomson Reuters). Descriptive univariate statistics were calculated (frequencies/ percentages) and displayed in graphs, as well as tables.
Results
Online Survey
Sixty residents, ninety fellows, and 184 faculty from the Department of Medical Imaging at the University of Toronto were invited to participate in the online survey and a 10-minute phone interview. A total of 117 residents, fellows, and faculty from a single Medical Imaging Department responded to the survey. Participants were given a list of professional experiences and they selected all the options with which they self-identified. The participants had varied professional experiences; 65.14%, 47.71%, 45.87%, 35.78%, 14.68%, and 8.26% of participants were a trainee (resident or fellow), co-principal investigator, staff or faculty, principal investigator, research assistant, and teaching assistant, respectively.
Previous experiences and software preferences. Based on responses to question #2 in the survey and as shown in Figure 1, participants had varying experiences including no experience (35.78%), a formal undergraduate course (27.52%), a formal postgraduate or graduate course (21.10%), or only an informal learning experience (15.60%). Informal learning experiences may include online learning. The most preferred software for participants was also diverse with a preference for SPSS (42.65%), Excel (27.94%), MATLAB (10.29%), SAS (7.35%), R (2.94%), STATA (2.94%), or other software (5.88%).
Content priorities for statistics and research methodology training. In response to question #4 in the survey, participants selected the top three content areas they felt were valuable to their career. ‘Research manuscript writing’, ‘use of statistics for estimation and hypothesis testing’ and ‘research proposal writing’ were selected as the first, second, and third most common choice.
In response to question #5 in the survey, participants also selected all the content areas about which they felt competent. ‘Research manuscript writing’, ‘use of statistics for estimation and hypothesis testing’ and ‘research proposal writing’ were selected by 66.0%, 26.0%, and 45.0% of participants, respectively.
Preferred learning formats. In response to question #6 in the survey, participants ranked different formats of learning according to their preference for how to receive training in statistics and research methodology. A graph depicting how many participants selected learning formats as a first choice is depicted in Figure 2. Direct mentorship was the most popular first choice option. Inperson lectures were the second most popular first choice, closely followed by workshops.
Interviews
To augment the online surveys, five semi-structured tenminute interviews were conducted to gain insights on the reasons behind their preferred learning format, as well as feelings towards their current level of statistics and research methodology knowledge. Since these are qualitative results, the quantitative significance of the interviews is not relevant. Participants were also asked how they thought a statistics and research methodology training program would impact the department.
Current level of statistics and research methodology knowledge. All the interviewees had different previous experiences with statistics and research methodology, but they all felt their knowledge could be improved. Key concerns expressed were: (1) uncertainty about when to perform different procedures, (2) the importance of more related practice, and (3) lack of confidence. It was frequently mentioned that an expert in statistics is usually hired for research projects, rendering it irrelevant to know procedures in-depth. Instead, knowing how to choose different statistical procedures was identified as important. It was noted that the lack of a preference for the SAS and R programs was related to limited knowledge and experience with the software.
Preferred learning format. Four interviewees selected ‘direct mentorship’ as their preferred learning format. Key constructs expressed were: (1) interactive experience, (2) individualized learning experience, (4) comfortable environment, and (5) convenience. Analysis indicated that data saturation had been reached, as the final two interviews did not result in new information. One interviewee selected ‘in-person lecture’ as their preferred learning format. Key constructs were: (1) familiarity with learning format and (2) comfort.
Impact of a training program. All interviewees agreed that a statistics and research methodology training program would be beneficial to the Medical Imaging Department at the University of Toronto. Key constructs expressed were: (1) improved ability to interpret research and (2) improved ability to work with experts on research projects.
Review of Departmental Publications
The review identified 322 articles published in 2013 by 177 faculty members of the Department of Medical Imaging at U of T, including 26 full professors, 43 associate professors, 92 assistant professors, and 16 lecturers. The 322 articles were randomized, and a representative sample of 170 articles were reviewed. The most common study types were retrospective cohort (22%), prospective cohort (17%), review (15%), and case-report (14%) (see Figure S6).
Depth of statistics knowledge utilized. Figure 3 shows that 110 (65%) of the articles analyzed used statistical methods. After recording the type of statistical analysis for each of the reviewed articles, the level of statistics that an article contained was determined. The level categories are described in Figure S5. The most common level of statistics used was Basic I, which was the case for 49 (44%) of the articles analyzed. The second most common level of statistics used was Basic II, which was the case for 37 (34%) of the articles analyzed. 21 (19%) of the studies analyzed required a level of statistics that would be classified as Intermediate. Only 3 (3%) of the studies analyzed required a level of statistics that would be classified as Advanced.
Impact of research. In terms of the journal articles published, 82.0% of the articles had an impact factor of 0-5. The median journal impact factor was 2.7 (IQR = 1.7 – 3.7). The three most common frequent journals to publish in were the American Journal of Neuroradiology, Pediatric Radiology Journal, and Canadian Association of Radiologists Journal, in which six, four, and three percent of articles were published, respectively. Results from a previous study conducted by Tyrrell et al. found that the median h-index of faculty in the department for the year 2013 was 10 (IQR = 4 – 22). Only 20% of corresponding authors were faculty from within the department [10].
Discussion
Medical imaging trainees and clinicians have unique statistics and research methodology training needs, as well as pedagogical preferences. These data support the continued need to offer broad training in research methodology, but suggest that a more cost effective and enabling education program would result from emphasis on basic level statistics. It was found that: almost half of the participants in the survey had no experience or only informal experience with statistics and research methodology; knowledge gaps existed, especially in the ‘use of statistics for estimation and hypothesis testing’. Gaps in statistics and research methodology preparation for medical imaging trainees and clinicians exist, illustrating not only the importance of an evidence-based training program, but also specific content recommendations.
A baseline level of knowledge with regards to statistics and research methodology is important. The ability to ‘work with experts who have a background in statistics on research projects’ was highlighted as particularly important in interviews. The ability to communicate with experts was found to be valuable, rather than the capacity to complete analyses. To meet these needs, a training program that focuses on a basic level of statistics is valuable, rather than in-depth knowledge of advanced statistical procedures.
Medical trainees and clinicians have learning style preferences. Expectedly, direct mentorship and workshops were preferred, noting that they are pedagogical formats that provide an ‘interactive experience’, an ‘individualized learning experience’, a ‘comfortable environment’ and ‘convenience’. Tailoring the learning format of statistics and research methodology training programs may be beneficial; aligning teaching methods to participants’ learning styles and preferences improves engagement, as well as success [11]. Surprisingly, in-person lectures were also preferred, a counterintuitive finding given the current trend towards interactive teaching methods. Reasons in interview responses for preferring in-person lectures included ‘familiarity with learning format’ and ‘comfort’. This may be linked to didactic learning being a dominant form of instruction in the field of medical imaging already [12]. The interview and survey findings both supported the importance of research methodology and statistics training for medical imaging trainees and clinicians.
Review of Department Publications
The review of published articles elucidated needs of the department. Most published articles were retrospective or prospective cohorts, and the level of statistics employed in most of them was at a basic level. This implies that instruction on basic statistics would be highly beneficial. Only a small percentage of the research from the department was published in high impact factor journals, and that the median h-index of the corresponding authors was relatively low. These results emphasized the value of fewer but impactful publications, rather than numerous low-impact publications [10]. Since high impact journals typically have a wider readership, publishing in these journals may increase the dissemination of research findings. Therefore, with the aim of maximizing the impact of articles, researchers need to consider ways to improve the quality of their publications, such as incorporating more advanced statistical analysis. However, it is important to keep in mind that journals with relatively higher impact factors do not necessarily have higher statistical standards, which has been shown in a study reporting the wide variability in statistical practices across journals with high or low impact factors [13]. Regardless, implementing a research methodology training program may increase researchers’ knowledge of the different statistical methods available for analyzing and interpreting their data, thereby enhancing their studies’ methodological quality and potentially increasing the chances of the studies being published in a journal with a high impact factor. Indeed, a dedicated resident research program has previously increased quantity and quality of publications by orthopaedic residents [14].
Limitations
Due to the nature of self-reported data, the findings of the study cannot be generalized. There are limitations in self-reported accounts of personal experiences and skills, as well as the determination of what learning environment would be best for the partcipants [15]. The study only included participants from the Department of Medical Imaging at the University of Toronto, and the review was conducted over a randomized sample of the department’s publications over one year. Furthermore, characterization of the use of statistics in the department may have been more accurate if a review of all published articles was conducted instead. Findings from this study may not reflect other institutions.
Future work
The findings of this study will be used for the research team to support the design of statistics and research methodology training programs at our department. In response to a preference for learning through direct mentorship, efficient and creative teaching through a mentored learning environment will be emphasized. Opportunities that are tailored to participants’ professional responsibilities, concentrate on statistics for medical research and medical image feature analysis, and include research project oversight, will be better integrated.
Conclusion
Academic departments need to address their own training needs in order to better support the development of research methodology curricula. Results from this study support the need to offer broad training in research methodology and suggest that a more cost effective and enabling education program would result from better emphasis on basic level statistics.
References
- 1. General Medical Council. Tomorrow’s Doctors: Outcomes and Standards for Undergraduate Medical Education. (General Medical Council, 2009).
- 2. Fielding, S., Poobalan, A., Prescott, G. J., Marais, D. & Aucott, L. Views of medical students: what, when and how do they want statistics taught? Scott Med J 60, 164–169 (2015).
- 3. Supino, P. G. & Richardson, L. D. Assessing research methodology training needs in emergency medicine. Acad Emerg Med 6, 280–285 (1999).
- 4. McCorry, L. K. & Mason, J. Chapter 3: Verbal communication. in Communication Skills for the Healthcare Professional 34–53 (Lippincott Williams & Wilkins, 2011).
- 5. Gore, A., Kadam, Y., Chavan, P. & Dhumale, G. Application of biostatistics in research by teaching faculty and final-year postgraduate students in colleges of modern medicine: A crosssectional study. Int J Appl Basic Med Res 2, 11–16 (2012).
- 6. Hannigan, A., Hegarty, A. C. & McGrath, D. Attitudes towards statistics of graduate entry medical students: The role of prior learning experiences. BMC Med Educ 14, 70 (2014).
- 7. Barbui, C., Addis, A., Amato, L., Traversa, G. & Garattini, S. Can systematic reviews contribute to regulatory decisions? Eur J Clin Pharmacol 73, 507–509 (2017).
- 8. Creswell, J. W. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. (SAGE Publications, 2009).
- 9. Braun, V. & Clarke, V. Using thematic analysis in psychology. Qual Res Psychol 3, 77–101 (2006).
- 10. Tyrrell, P. N., Moody, A. R., Moody, J. O. C. & Ghiam, N. Departmental h-Index: Evidence for Publishing Less? Can Assoc Radiol J 68, 10–15 (2017).
- 11. Evans, N. J., Forney, D. S., Guido, F. M., Patton, L. D. & Renn, K. A. Chapter 8: Kolb’s theory of experiential learning. in Student Development in College: Theory, Research, and Practice 137–152 (Jossey-Bass, 2009).
- 12. Bedoya, M. A. et al. Learning, technology and intellectual property: A survey of the philosophies and preferences of our trainees and peers. Pediatr Radiol 46, 1780–1786 (2016).
- 13. Tressoldi, P. E., Giofré, D., Sella, F. & Cumming, G. High Impact = High Statistical Standards? Not Necessarily So. PLOS ONE 8, e56180 (2013).
- 14. Torres, D., Gugala, Z. & Lindsey, R. W. A dedicated research program increases the quantity and quality of orthopaedic resident publications. Clin Orthop Relat Res 473, 1515–1521 (2015).
- 15. Fielding, S., Poobalan, A., Prescott, G. J., Marais, D. & Aucott, L. Views of medical students: what, when and how do they want statistics taught? Scott Med J 60, 164–169 (2015
