P1: JDW
Zimmerman WL040/Bidgolio-Vol I WL040-Sample.cls June 20, 2003 17:20 Char Count= 0
516 USABILITYTESTING:ANEVALUATIONPROCESS FORINTERNETCOMMUNICATIONSData Analysis and Interpretation
Heuristic evaluations usually produce a list of the Web
site potential usability problems. The heuristic evaluation
clarifies what aspects of the site design and user behavior
clash and negatively impact the Web sites usability and
users’ experience. The list then provides Web site deve-
lopers with detailed information on specific aspects of the
Web site that need improvement.
Interpretation of an expert’s recommendations re-
volves around reviewing the guideline or principle or sug-
gestion and then figuring out how apply it to the Web
site. For example, guidelines suggest minimizing scrolling
on the home page. Often this requires reformatting the
page and reorganizing the basic structure with additional
branches.Verbal Protocol Analysis
Verbal, or “think aloud,” protocol analysis uses a set of
scenarios requiring participants to carry out tasks on the
Web site. As users work, they think aloud, explaining what
they are doing and why. Usability practitioners observe
and videotape participants, interpret the results, and re-
port problems to Web site developers with suggestions on
how to redesign the Web site to minimize the problems.
The scenarios and tasks reflect the intended use of the
Web site and may require participants to search for in-
formation, fill out forms, buy products online, conduct
simulations, or complete other Web site functions.
Verbal protocol analysis can be conducted in a usabil-
ity laboratory or in the field. Usability laboratories typi-
cally consist of waiting, observation, and test rooms. Test
rooms have personal computers connected to the Inter-
net with video cameras, microphones, and a one-way mir-
ror for observing participants. The observation room has
video mixing units, monitors, recording equipment, and
computers for logging observations and monitoring par-
ticipants in the adjacent test room. Usability practition-
ers often debate what is needed for a laboratory, and thus
some usability laboratories have a complete system and
others have only the most basic components—camera,
microphone, tripod, and video playback units. Usability
laboratories do not need all of the latest audio and video
recording and mixing technology to collect useful and
helpful data. In fact, usability practitioners can use a sin-
gle video camera, tripod, and microphone for collecting
data in the field. Other practitioners may have top-of-the-
line, portable usability laboratories with multiple cam-
eras, tripods, microphones, mixing and recording units,
laptop computers, and logging software.Methodology
Usability specialists create scenarios with tasks reflecting
the intended use of the Web site. Scenarios differ depend-
ing on the research question and objectives of the usability
testing. Some scenarios have participants use a particular
subarea or component of the Web site, others evaluate dif-
ferent page design and elements, and still others have par-
ticipants navigate the depth and breadth of the Web site.
Although usability testing is often complete with indi-
viduals, some usability testing may be conducting with
teams of participants, and still others may be conductedusing several participants completing tasks in a labora-
tory at once. Participants reflecting the users of the Web
site are recruited to assist with the testing. See Integrat-
ing Usability Intro the Design Process for a discussion
on sampling and recruiting. Usability practitioners can
collect both qualitative and quantitative data of the ver-
bal protocol analysis session. To begin, usability practi-
tioners observe and record participants’ actions on obser-
vational logging forms, paying especially close attention
to the problems participants encounter and how partici-
pants try to solve the problems.
Usability practitioners also note the beginning time
when a participants begins a task, as well as the begin-
ning time when a participant first encounters a problem
and the ending time, if the participant solves the prob-
lem. They also use advanced video systems using multi-
ple cameras can capture participants’ facial expressions,
body movements, and screen images. Some systems video
capture cards export the Web site image to a videotaping
system. Other systems use software programs to capture
screen and log movements, keystrokes, and navigational
sequences. Participants may be asked to complete ques-
tionnaires before and after the protocol analysis. While
the participant is completing the questionnaire, the us-
ability practitioner should write a brief summary of his
or her observations, the major problems encountered, and
how data were collected.Data Analysis and Interpretation
Data analysis may be either qualitative or quantitative,
depending on the data collected. For qualitative data, us-
ability practitioners review their written notes, observa-
tional logs, videotapes, and audio comments to produce
a comprehensive listing of problems in site design, lay-
out, navigation, navigational aides, site structure, content,
and readability. For quantitative data, the usability prac-
titioner summarizes the data, usually by entering the data
into statistical analysis programs and then generates sum-
mary and inferential statistics, when needed. Although the
list will not resolve the problems, it provides Web site de-
velopers with the information they need to improve the
site.
Interpreting the findings usually requires translating
the findings into strategies to solve a particular problem.
Thus, it requires trying to figure out what the underlying
cause of the problem might be and then considering the
alternative strategies for solving the problem and making
recommendations. For example, should participants re-
port that the download time is too long for a page, then us-
ability practitioners should determine the file size and the
speed of their connection mode and recommend strate-
gies for reducing the file size, such as reducing the visuals
file sizes.ETHICAL AND LEGAL
CONSIDERATIONS
When conducting usability evaluations, usability practi-
tioners should not develop any usability tests that would
put participants at risk. Usability practitioners should ask,
“Is there any way that an individual might be at physical,
psychological, or social risks by being involved in the