Case Studies in Knowledge Management

Learning from Simple Systems: The Case of JPL 101 9

Copyright © 2005, Idea Group Inc. Copying or distributing in print or electronic forms without written

we were looking at a four-week contest, consisting of four quizzes at 25 questions each,
and that it was highly unlikely that we would be able to identify a single winner based
on this design.

How Do You Determine the Winner?

One way to work around the inability to reliably create a single winner is to create
multiple categories of winners. We assumed that it would be harder for newer employees
than more experienced employees, and that different categories of employees would
shine in different subject areas. Based on these assumptions, participants would be
grouped based on number of years of tenure, with three categories of under five years,
five to 20 years, and more than 20 years, driven by the demographics of the Laboratory
and a desire for fair groupings.
A multitiered approach was chosen, with weekly results feeding into identification
of grand winners. The weekly results would be based on a score computed as the number
of right answers minus a fraction of the number of wrong answers, similar to the Scholastic
Aptitude Tests (SATs). Options for handling ties, which were highly likely on a weekly
basis, were a tie-breaker quiz, drawing names from the pool of highest scores, or simply

Quiz Design

Issue: Differentiate

~8000 potential

participants

Constraints:

• Keep time to 10-15 minutes


• Reasonable number of questions for KC team to develop


• Fairness

Results:

• 4 quizzes


• 25 questions per quiz


• Score based on number
  right/wrong


• Don’t use time tag as
  component of score


• Avoid negative scores by
  including 5 easy questions


• Unlikely to identify winner
  based solely on score

Base score

Options:

• Number right


• Number right – ¼ *
  number wrong
  Issues:


• Avoid negative scores


• Keep number
  challenging without
  being burdensome

Time score

Issues:

• Network latency


• Network loading


• Server loading

Info from Beta test results:

• 15 questions in 10 minutes
  is comfortable


• 25 questions in 10 minutes
  is challenging

Figure 4. Quiz design decision map