101has also been demonstrated to be a powerful resource in diagnostic challenges
involving high volume data processing, with crowdworkers successfully able to
identify colonic polyps [ 39 ] and red blood cells infected with malaria [ 40 ].
Crowdsourcing has even been used to generate epidemiologic symptom maps for
the flu, which have corresponded well to the Centers for Disease Control and
Prevention data [ 41 ].
Within the realm of clinical medicine, there has been significant interest in
crowdsourcing technology in the diagnosis and discrimination of disease processes.
In urology, large-scale evaluation has been used to help validate confocal laser
endomicroscopy as a potential technology for the diagnosis of urothelial carcinoma,
with crowdworkers able to correctly diagnose urothelial carcinoma of the bladder
92% of the time [ 42 ]. Within the field of ophthalmology, crowdworkers have been
used to identify abnormal fundi among diabetic patients and glaucomatous optic
disks, though sensitivity was high for both tasks, ranging from 83 to 100%, specific-
ity remained limited at 31–71% [ 43 , 44 ]. Additional refinement of crowdsourcing
for clinical diagnosis applications is clearly warranted, but this technology carries
enormous potential in the future of clinical medicine.
Recent research in medical education has also explored the potential of collec-
tive crowd wisdom to teach future generations of medical providers. A crowd-based
approach has been used to generate curricular content for trainees at both the pre-
medical and graduate medical education levels with initial success [ 45 , 46 ].
However, one of the most promising applications of crowdsourcing technology lies
in optimizing technical skills development for surgical trainees, which has been
demonstrated to be an area in particular need of innovation and adaptation for the
current training environment.
Crowdsourcing and Surgical Evaluation
One of the primary challenges in the current paradigm of surgical training is in pro-
viding, individualized, timely, and cost-efficient, feedback to trainees regarding
their technical skills. Moreover, the widespread trend toward simulation in surgical
education further generates a need for objective, formative feedback on a large scale
[ 1 ]. Indeed, simulation without feedback has been demonstrated to result in more
errors among trainees [ 14 ], suggesting that feedback is a critical part of simulation-
based learning. Given these demands, reliance on surgeon feedback alone becomes
difficult to sustain. The application of crowdsourcing to surgical skills evaluation
addresses these issues of efficiency, cost, and scalability.
In 2014, Chen and colleagues performed an initial study demonstrating the added
value of crowdsourced evaluation, which has provided a methodology upon which
most subsequent research on large-scale technical skills evaluation has been based.
Three groups were recruited to evaluate recorded robotic knot tying performance:
409 Amazon.com Mechanical Turk users, 67 Facebook users, and 9 faculty sur-
geons. Subjects first answered a qualification question by rating side-by-side videos
of Fundamentals of Laparoscopic Surgery peg transfer task performed by
6 Crowdsourcing and Large-Scale Evaluation