113bottom five on the match rank list, three of the five were identified by crowds, and
two of the five were identified by faculty to be among the bottom five performers in
these simulation tasks [ 56 ]. These findings suggest that technical performance alone
is not the primary predictor of success within surgical training. Indeed, there are
multiple other intangible factors assessed during an interview, such as tenacity, flex-
ibility, creativity, discipline, and emotional intelligence, which contribute to success
in residency training. However, this study further suggests that crowd-based skills
assessment may be particularly useful to identify those who are weak in surgical
skills and provides a framework by which crowdsourced evaluation could be used
to incorporate technical skills assessments into the complex process of resident
recruitment.
Though a critical component of surgical competency, excellent technical skill
alone clearly does not define excellent surgical care. This also involves complex
preoperative and intraoperative decision-making, surgical knowledge, and interper-
sonal patient care skills. These are more challenging to develop but are equally
essential parts of a comprehensive surgical education. With the recent adoption of
milestone-based competency for surgical training, there is increased focus on help-
ing trainees achieve competency in not only technical performance but also in the
realms of professionalism and communication, which are more difficult to objec-
tively measure. Surgical curricula directed toward these proficiencies have histori-
cally been evaluated using standardized patient interactions and the Objective
Structured Clinical Examination [ 68 , 69 ]. However, such evaluative measures carry
the same challenges of cost, time investment, and subjectivity inherent to surgical
skills feedback. To date, large-scale evaluation of such competencies has not yet
been explored, but integration with these existing assessment tools might be a novel
application of this technology. The use of crowdsourcing to enhance such nontech-
nical aspects of surgical education remains a potential area of new research.
Conclusions
The use of crowd-based evaluation in the realm of surgical education remains
relatively novel, but the present literature has demonstrated that this technology
may address a significant challenge in the current paradigm of surgical educa-
tion. The consistency, economics, and rapidity of crowd-based feedback may be
one means by which the attainment of technical proficiency among surgical
trainees can be facilitated in the face of current work-hour limitations, adminis-
trative burdens, and inconsistent feedback mechanisms that constrain residency
training programs across multiple surgical specialties. As new surgical technolo-
gies and techniques continue to develop, crowdsourcing may also prove to be an
integral part of continuing medical education and skills development for those
surgeons already in practice. Moreover, given the initial promise of this technol-
ogy within the realm of technical skills development, crowd-based evaluation
may soon become even more broadly used to help trainees develop nontechnical
skills, validate national surgical curricula, and inform the selection of future
surgical trainees.6 Crowdsourcing and Large-Scale Evaluation