PD30-05 WISDOM OF THE CROWDS: USE OF CROWDSOURCING TO ASSESS SURGICAL SKILL OF ROBOT-ASSISTED RADICAL PROSTATECTOMY IN A STATEWIDE SURGICAL COLLABORATIVE

摘要

You have accessJournal of UrologyProstate Cancer: Localized II1 Apr 2015PD30-05 WISDOM OF THE CROWDS: USE OF CROWDSOURCING TO ASSESS SURGICAL SKILL OF ROBOT-ASSISTED RADICAL PROSTATECTOMY IN A STATEWIDE SURGICAL COLLABORATIVE James Peabody, David Miller, Brian Lane, Richard Sarle, Andrew Brachulis, Susan Linsell, Tae-Kyung Kim, Deepansh Dalela, Mani Menon, James Montie, Bryan Comstock, Tom Lendvay, and Khurshid Ghani James PeabodyJames Peabody More articles by this author , David MillerDavid Miller More articles by this author , Brian LaneBrian Lane More articles by this author , Richard SarleRichard Sarle More articles by this author , Andrew BrachulisAndrew Brachulis More articles by this author , Susan LinsellSusan Linsell More articles by this author , Tae-Kyung KimTae-Kyung Kim More articles by this author , Deepansh DalelaDeepansh Dalela More articles by this author , Mani MenonMani Menon More articles by this author , James MontieJames Montie More articles by this author , Bryan ComstockBryan Comstock More articles by this author , Tom LendvayTom Lendvay More articles by this author , and Khurshid GhaniKhurshid Ghani More articles by this author View All Author Informationhttps://doi.org/10.1016/j.juro.2015.02.1826AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookTwitterLinked InEmail INTRODUCTION AND OBJECTIVES Because a surgeon's technical skill may be a pivotal determinant of outcomes following robotic-assisted radical prostatectomy (RARP), the Michigan Urological Surgery Improvement Collaborative (MUSIC) examined the degree to which expert and crowd sourced reviewers can distinguish differences in performance among fully-trained urologists. METHODS MUSIC is a statewide consortium of 42 urology practices. Twelve MUSIC surgeons submitted a video of a representative RARP which were reviewed by 25 surgical experts from MUSIC and crowdsourced reviewers from the Amazon Mechanical Turk platform using the Crowd-Sourced Assessment of Technical Skills (C-SATS) on-line survey tool. Four key steps (bladder neck division, nerve sparing, apical dissection, and urethrovesical anastomosis) from each video were edited into 76 video-clips (maximum duration 10 minutes) for analysis using the Global Evaluation Assessment of Robotic Skills (GEARS) instrument. Unedited complete anastomosis clips from 8/12 surgeons (range 9-37 minutes) were also assessed using the Robotic Anastomosis Competency Evaluation (RACE) instrument. Inter-rater reliability was evaluated using Krippendorff's alpha and intraclass correlation coefficient (ICC). Linear mixed-effects models derived mean crowd and expert ratings for each video-clip. Individual video-clip ratings were aggregated to provide surgeon scores. RESULTS MUSIC experts completed 318 video ratings in 15 days; crowd-workers completed 2,531 ratings in 21 hours. Expert GEARS ratings lacked sufficient internal consistency across videos (Krippendorff's alpha=0.25, ICC=0.24) to validate against crowd-workers. The correlation between crowd and expert scores was higher (r=0.78) when using aggregated surgeon scores (Figure a). For the RACE instrument, inter-rater reliability was better (ICC=0.55) and Pearson correlation with crowd scores was 0.74 (Figure b). For both rating tools, experts and the crowd consistently agreed in the rank order of the lower scoring surgeons. CONCLUSIONS Expert surgeons and the crowd demonstrate high levels of agreement for identifying lower scoring RARP surgeons. The ability of crowds to provide rapid reviews of surgical videos suggests a potential role for crowdsourced methodology in emerging training and quality improvement initiatives of surgical performance. © 2015 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetailsCited byChen J, Cheng N, Cacciamani G, Oh P, Lin-Brande M, Remulla D, Gill I and Hung A (2018) Objective Assessment of Robotic Surgical Technical Skill: A Systematic ReviewJo