MP11-08 CROWDSOURCING ASSESSMENT OF SURGEON DISSECTION OF RENAL ARTERY AND VEIN DURING ROBOTIC PARTIAL NEPHRECTOMY: A NOVEL APPROACH FOR QUANTITATIVE ASSESSMENT OF SURGICAL PERFORMANCE

Abstract

You have accessJournal of UrologySurgical Technology & Simulation: Training & Skills Assessment I1 Apr 2016MP11-08 CROWDSOURCING ASSESSMENT OF SURGEON DISSECTION OF RENAL ARTERY AND VEIN DURING ROBOTIC PARTIAL NEPHRECTOMY: A NOVEL APPROACH FOR QUANTITATIVE ASSESSMENT OF SURGICAL PERFORMANCE Mary Powers, Thomas Lendvay, Aaron Boonjindasup, Michael Pinsky, Philip Dorsey, Michael Maddox, Bryan Comstock, Li-Ming Su, Matthew Gettman, Chandra Sundaram, Erik Castle, Jason Lee, and Benjamin Lee Mary PowersMary Powers More articles by this author , Thomas LendvayThomas Lendvay More articles by this author , Aaron BoonjindasupAaron Boonjindasup More articles by this author , Michael PinskyMichael Pinsky More articles by this author , Philip DorseyPhilip Dorsey More articles by this author , Michael MaddoxMichael Maddox More articles by this author , Bryan ComstockBryan Comstock More articles by this author , Li-Ming SuLi-Ming Su More articles by this author , Matthew GettmanMatthew Gettman More articles by this author , Chandra SundaramChandra Sundaram More articles by this author , Erik CastleErik Castle More articles by this author , Jason LeeJason Lee More articles by this author , and Benjamin LeeBenjamin Lee More articles by this author View All Author Informationhttps://doi.org/10.1016/j.juro.2016.02.2379AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookTwitterLinked InEmail INTRODUCTION AND OBJECTIVES We sought to describe a methodology of crowdsourcing for obtaining quantitative performance ratings of surgeons performing renal artery and vein dissection of robotic partial nephrectomy (RPN). We sought to compare assessment of technical performance obtained from the crowdsourcers to those of surgical content experts (CE). Our hypothesis is that the crowd can score performances of renal hilar dissection comparably to surgical CE using the Global Evaluative Assessment of Robotic Skills (GEARS). METHODS A group of resident and attending robotic surgeons (n= 5) submitted a total of 14 video clips of RPN during hilar. These videos were rated by both crowd and CE for technical skills performance using GEARS. A minimum of 3 CE and 30 Amazon Mechanical Turk crowdworkers evaluated each video with the GEARS scale. RESULTS Within 13 days we received ratings on all videos from the 3 CE and within 11.5 hours we received 548 GEARS ratings from crowdworkers. Even though CE were exposed to a training module, internal-consistency across videos of CE GEARS ratings remained low (ICC=0.38). Despite this, we found that crowdworker GEARS ratings of videos were highly correlated with CE ratings at both the video-level (R=0.82, p<0.001) and surgeon level (R=0.84, p<0.001). Similarly, crowdworker ratings of the renal artery dissection were highly correlated with expert assessments (R=0.83, p<0.001) for the unique surgery-specific assessment question. CONCLUSIONS We conclude that crowdsourced assessment of quantitative performance ratings may be a suitable alternative to surgical experts ratings and would provide a rapid, scalable solution to triage technical skills. © 2016FiguresReferencesRelatedDetails Volume 195Issue 4SApril 2016Page: e114 Advertisement Copyright & Permissions© 2016MetricsAuthor Information Mary Powers More articles by this author Thomas Lendvay More articles by this author Aaron Boonjindasup More articles by this author Michael Pinsky More articles by this author Philip Dorsey More articles by this author Michael Maddox More articles by this author Bryan Comstock More articles by this author Li-Ming Su More articles by this author Matthew Gettman More articles by this author Chandra Sundaram More articles by this author Erik Castle More articles by this author Jason Lee More articles by this author Benjamin Lee More articles by this author Expand All Advertisement Advertisement PDF downloadLoading ...