The Internet Encyclopedia (Volume 3)

P1: JDW

Zimmerman WL040/Bidgolio-Vol I WL040-Sample.cls June 20, 2003 17:20 Char Count= 0

520 USABILITYTESTING:ANEVALUATIONPROCESS FORINTERNETCOMMUNICATIONS

Collect Data

Whenever possible, use multiple data collection meth-

ods to minimize the chances of drawing erroneous con-

clusions. Verbal protocol analyses often include direct

observations of participants, videotaping their actions,

recording their comments, and having them complete

questionnaires. When participants arrive for testing, pro-

vide a brief overview of the evaluation, but do not make

judgmental statements that may bias the test results.

Brief participants on all forms, have them complete the

forms, and give them the honorarium. Once they have

signed the forms and completed pre-questionnaires, ac-

company participants to test rooms. Orient participants

to the equipment and Web site they will be evaluating,

give them the scenarios, and review the scenarios with

them.

When videotaping or audio taping participants’ com-

ments, make sure the video cameras and audio equipment

work properly and that you have sufficient tape to com-

plete the session. Brief participants on their tasks, and

remind them to think aloud as they work. As participants

work, you may need to remind them to keep talking and

telling you what they are thinking.

While participants are working, remain as unobtrusive

as possible, yet remain available to participants if they

need help or become anxious. Use observational logs (see

Figure 2) to record your observations and notes of the

problems they encounter. Should a participant become

nervous or irritated, stop the testing. After participants

have left each verbal protocol analyses session, write a

succinct summary of the test session, your observations,

the participant’s response to testing, and the problems the

participant encountered.

Analyze Data

Analyzing data can involve both qualitative and quantita-

tive tasks. For example, textual notes from exit interviews

require qualitative analysis. Qualitative analysis requires

reading through the responses, familiarizing oneself

with the responses, and then summarizing the findings

verbally. On the other hand, questionnaires with set

responses generate quantitative data. Summarizing re-

sponses to such questions requires entering the data into a

spreadsheet or statistical analyses program, such as SPSS

or SAS, for statistical analyses.

Use triangulation to validate findings by comparing

data from different methodologies. For example, a re-

searcher may validate data about a site’s navigational

aides from his or her observation notes, screen tracking

software, open-ended questions, and the usability ques-

tionnaire.

Interpret the Data

Consider prioritizing the severity of the problems. Re-

sources, staff members, and the time available to cor-

rect the problems are often limited, so many organiza-

tions must rank order the problems and correct only the

more serious or critical ones or those that will affect the

largest number of users. Severity ratings can be based on

the time required to complete a task, the number of users

who might encounter the problem, the negative impact on

Usability Observation Form Page Number _____

Participant No. _________________________

Observational Sheet _________________________

Date: _________________________

Observer: _________________________

Search engine _________________________

Task

Number

Beginning

Time

End

Time

Problem/ how resolved

Figure 2: Observational sheet for recording participants actions when carrying out

tasks.