Virtual Reality to Inform and Facilitate Trainee Assessment

摘要

Virtual reality allows the purposeful design of interactions for establishing validity evidence and conducting standard setting procedures for performance assessments. It has been employed for traditionally challenging to assess content, including clinical deterioration, behavioral health, conflict management, and procedural training. In our study “Establishing Objective Measures of Clinical Competence in Undergraduate Medical Education Through Immersive Virtual Reality”, we explored the feasibility of immersive Virtual Reality – a virtual 3D environment where learners interact with digital content - to establish objective standards for performance assessment.1Zackoff M.W. Young D. Sahay R.D. Fei L. Real F.J. Guiot A. Lehmann C. Klein M. Establishing Objective Measures of Clinical Competence in Undergraduate Medical Education Through Immersive Virtual Reality.Acad Pediatr. 2021 Apr; 21: 575-579Abstract Full Text Full Text PDF Scopus (0) Google Scholar We identified observable behaviors that predicted competency ratings for identification of impending respiratory failure. In this cross-sectional observational study of fourth year medical students, the borderline groups method for standard setting was used to identify specific observed behaviors (e.g., indicating a need for escalation of care, commenting on abnormal vital signs) that predicted a global rating of competent (vs. borderline or not competent). Evaluators can leverage these observable behaviors to form the basis of an assessment metric that may predict, and potentially replace, subjective global assessments of competency to assess an infant with respiratory distress. The purpose of this progress report is to highlight recent uses of virtual reality for establishing or conducting trainee assessments since we submitted our original manuscript on June 10th, 2020. Our original work informed a new study leveraging a direct observation instrument to assess the impact of COVID-19 on a trainee’s ability to identify signs of impending respiratory failure.2Odum J.D. Real F.J. Rice J. Meisman A. Sahay R. Zhang B. Zackoff M.W. Virtual Reality to Assess Resident Recognition of Impending Respiratory Failure During COVID-19.Hosp Pediatr. 2023 May 26; e2022006917Crossref Scopus (0) Google Scholar Due to the COVID-19 pandemic, trainees had decreased clinical volumes with less exposure to respiratory failure. We used the previously tested virtual reality simulations as an assessment simulation with pediatric residents. We found that only 23% of residents indicated that the patient in impending respiratory failure required an escalation of care, elucidating a key educational gap following a period of significantly reduced clinical exposure. Virtual reality was crucial for not only establishing the observable behaviors associated with competency, but also facilitated assessment of trainee performance in a consistent manner that would otherwise not have been feasible. Virtual reality has been used to support competency-based assessment of learners’ communication skills. Using 42 virtual reality simulation observations, investigators developed a novel instrument to assess pediatric residents’ behavioral health anticipatory guidance and motivational interviewing skills.4Real F.J. Whitehead M. Rosen B.L. Meisman A. Crosby L.E. Klein M.D. Herbst R.B. Assessment of Competency-Based Behavioral Health Anticipatory Guidance Skills Among Pediatric Residents: The Role of Virtual Reality.J technol behav sci. 2022; : 115-124Crossref Scopus (3) Google Scholar The recorded simulations captured learners’ real-time interactions with virtual characters over three scaffolded scenarios standardized for verbal and non-verbal responses to specific counseling behaviors. These observations elucidated a wide range of learners’ behaviors in support of establishing validity evidence (anchored in Kane’s four stage model3Cook David Brydges Ryan Ginsburg Shiphra Hatala Rose A contemporary approach to validity argume