111One of the limitations of the feedback provided by crowdworkers is that it is
largely reflective rather than formative. Most of the published studies on crowd-
sourced feedback for technical skills asked crowds to evaluate the performance of a
specific task using an objective numeric scoring system, rather than to provide a
subjective critique. In contrast, subjective feedback from expert surgeons might be
not only reflective but also corrective, thereby facilitating refinement in surgical
technique. Even in studies where subjective feedback was solicited from crowd-
workers, the potential utilization of such comments to generate subtle improve-
ments in technique was not specifically examined [ 47 , 54 ]. It is notable, however,
that subjective evaluation of expert and crowd comments regarding the same task
performance reveal that the two encompass similar content, suggesting that perhaps
either form of feedback might be of similar value to the trainee [ 54 ]. Moreover,
there is evidence to suggest that specific and individualized feedback may not be as
critical for adequate technical skills development as previously believed, particu-
larly among novice learners [ 62 ].
Though most of the literature on crowdsourced feedback focuses on refining the
technical skills of surgical trainees, emerging work has suggested that this technol-
ogy might be further applied to refine the skills of those already considered to be
“expert surgeons.” Even attending surgeons and surgeons in practice are not imper-
vious to error. Indeed in a study of national malpractice claims, 58% of technical
errors resulting in patient harm involved a surgeon practicing within his or her own
specialty but lacking in technical expertise [ 6 ]. Moreover, whether it is to maintain
a pre-existing skill set or to become proficient with new surgical technology, even
experienced surgeons will need to develop and hone their surgical skills throughout
the course of their careers.
Ghani and colleagues studied the use of crowd-based evaluation through the lens
of quality improvement in a population of practicing urologists through the Michigan
Urological Surgery Improvement Collaborative (MUSIC). Overall, 76 video clips
of technically challenging portions of nerve-sparing robotic-assisted laparoscopic
prostatectomy from 12 surgeons within the consortium were selected for evaluation
by at least 4 surgical experts and at least 30–55 Amazon Mechanical Turk workers
per clip using the GEARS assessment and the Robotic Anastomosis and Competency
Evaluation (RACE) for urethrovesical anastomosis video segments. Both GEARS
and RACE scores between the two groups were strongly and significantly correlated
(Pearson’s correlation = 0.78 and 0.74, respectively; p < 0.001). There was signifi-
cantly greater intra-peer variability in ratings from expert surgeons (p < 0.001).
Expert peer reviewers took 15 days to return both global skills and anastomosis
ratings, whereas crowdworkers returned global ratings on average in 21 h and anas-
tomosis ratings in 38 h. Moreover, both the crowdworkers and experts were able to
identify the bottom five surgeons ranked by technical skill for this procedure [ 53 ].
The use of crowdsourced feedback may therefore be valuable for experienced sur-
geons as well, as it may provide a model for continued surgical skills refinement,
facilitate future peer evaluation of currently practicing surgeons, and lend itself to
quality improvement initiatives in practice.
6 Crowdsourcing and Large-Scale Evaluation