Evaluation: Think-Aloud and Cooperative Evaluation

Think-Aloud and Cooperative Evaluation

Think-Aloud

Another common evaluation technique is "think-aloud", in which the user performs a number of tasks and is asked to think aloud to explain what they are doing at each stage, and why. The evaluator records the users actions (sometimes using tape recordings or video or computer logging or via user notes - here we'll use one of the most popular forms, pencil and paper) for each of the tasks, as well as noting any problems. Think-aloud has the advantage of simplicity, it requires little expertise to perform, and can provide a useful insight into the problems with an interface. However, there are a number of problems too. The information is necessarily subjective, and can be selective depending on the tasks chosen. Being observed and having to describe what you are doing can also affect the way in which you do something: ask a juggler to describe how she juggles.....

Cooperative evaluation

"Cooperative evaluation" is a variant of think aloud, in which the user is encouraged to see himself as a collaborator in the evaluation rather than just a subject. As well as getting the user to think aloud, the evaluator can ask such questions as "Why?" and "What if.....?"; likewise, the user can ask the evaluator for clarification if problems arise. This more relaxed approach has a number of advantages. It is less constrained and therefore easier for the evaluator, who is not forced to sit in solemn silence; the user is encouraged to actively criticise the system rather than simply suffer it; and the evaluator can clarify points of confusion so maximising the effectiveness of the approach. Note that it is often not the designer who is the evaluator, but an independent person.

Problems

One of the problems with both these techniques is that they generate a large volume of information which has to be painstakingly and time-consumingly analysed. Such a record of an evaluation session is known as a protocol, and there are a number to use; pen and paper, audio and video recording, computer logging and user diaries. Analysis can be partially automated for some scenarios, but for the pen and paper method, the protocol analysis has to be done by hand.

Think-Aloud and Cooperative Evaluation: practical

Take it in turns to be an evaluator (of your own system) and a user (of another system - preferably a different topic). As an evaluator spend a few minutes thinking of some scenarios and tasks for the user to perform. Include some complex tasks as well as some simple ones. Decide on whether you are going to use think-aloud or cooperative evaluation. Then run the evaluation, keeping full notes of the users actions and behaviour and any problems. The fuller the detail in the protocol, the better.

As a user, follow the evaluator's instructions. Try to give as much information as possible. People tend to say less when they are unsure what to do, but this is the time that the evaluator needs to know most. Go through explaining which buttons you would press and when, giving the designer chance to show you what, if anything, would happen to the display (if you are using cooperative evaluation, you can discuss it with them, but for think aloud you have to just accept what is presented).