Is There an Art to Usability Test Moderation?

Recently, I have been rapidly teaching non-user experience professionals about usability studies and eye tracking studies. Due to the limited time, I have really been focusing on the tools and have not had the chance to delve deeper into the exact art of moderation. And in my opinion, moderating usability and eye-tracking tests is a skill that can only be perfected in time. You can teach someone a tool to generate data, but it takes more effort and experience to teach someone how to moderate. Good moderators know when to probe, when to be quiet, and when to look beyond verbal communication.

Being a good moderator also means knowing when to use think-aloud protocol. Think aloud protocol was first seen at IBM, and then refined by Ericsson and Simon. Most versions of think aloud protocol we use today is a variation of the Ericsson and Simon version where there is more active listening and more of a dialogue between moderator and participants. The more recent version encourages the use of certain acknowledgement tokens, such as “Mm hmm” rather than Ericsson and Simon’s token of “keep talking”. Some acknowledgement tokens can influence changes in speakership between the participant and the moderator.

When using think aloud protocol you have to be able to explain to the participant how to think aloud and keep reminding him or her to do so throughout the session. There are instances where you may not want to think aloud – eye-tracking studies or formative studies where you are concerned with just finding out critical problems that prevent users from completing their tasks. This past Monday, at my HF750 Testing and Assessment class at Bentley, we watched sessions of usability tests and analyzed moderators. We even watched a famous author of a usability book moderate a remote test and observed mistakes he made. It is clear that it takes years of experience, and even then, the experts can make mistakes.

Moderation is so important because the better the moderation, the better the data. Isn’t getting accurate data the point of testing? People are easily influenced, so moderator bias is a real concern. The way you ask the questions is crucial and the vibe you give off is equally as important.

Tweet