It’s hard to ask questions that don’t suck when you’re running a study, so the best advice is “don’t do it.” But you are going to anyway, so make sure your questions are grounded in what the user has done, not what you want them to do.
It would be much easier if you could just see inside your users’ heads while they are working with your product. You might think that asking them questions will give you extra insight, but instead it’s likely to divert things down the path of your choosing.
Sesame Street’s number of the day: 10
You’re proud of your work, you’re pretty good at what you do, and you’ve spent some time crafting what you think is the perfect interface.
Now you’ve got a user working through a task with the UI and they are struggling. What’s the first thing you want to ask?
The correct answer, most of the time, is NOTHING. You don’t want to ask a question at all. The first thing you want to do is count to ten slowly in your head.
Counting to ten gives you time to let the user figure things out for themselves. Your question was probably going to give them hints about what they should be doing – like “Did you see the big red Submit button?” Your question was probably also going to interrupt their thought processes.
Remember – this is the first time they have seen the interface. It’s going to take them longer to figure it out than you expect.
Counting to ten also gives you time to formulate a good question. If you are using a think-aloud protocol then all you probably want to say is “Please remember to think out loud.” If you really, really have to ask a question, and it can’t wait until the end of the session, then you need to put some thought into exactly what you’re going to say.
Sesame Street’s word of the day: Humility
If there’s one word you need to keep in mind during a user session, it’s humility. Think of it like this: mentally prepend the words “I’m sorry I’m such a poor developer, but could you tell me…” to your question. Now, if your question still seems valid, it might be worth asking.
Any time you ask a question during a user session you are pulling the participant in a different direction than they had intended. Even if they are thinking out loud for you, it’s hard to know where they’ll go next. Talking to them pulls them away from their task and back into the artificial world of your usability test.
The only way to minimize this diversion is to ask a question about something the user did, not about something you wish they’d done. It’s easy to kid yourself, so here’s a tip: Ask a question about something you just observed, and then leave the user hanging. For instance “I notice that you just entered your e-mail address and then deleted it again… [silence]” or “Just now you searched for Fubars, but you didn’t follow any of the results… [silence].“
You will find that as long as you stop talking and leave a little silence, the participant will start talking about their behavior, elaborate on it a bit, and then keep working on their task. Normally this won’t divert them to the point where they need time to re-orient themselves back into the task after they’ve answered the question.
Save your questions until the end
Of course, any interruption is likely to divert the participant. It will take them time to get back into the flow of things, and they’ll be wondering why you asked the question – were they doing something wrong? Should they be using the feature you asked about, even if they hadn’t intended to?
The best way to prevent this is to write your questions down and save them until the end of the session. By then, you may find that you’ve got answers to most of them anyway. If not, you can use the open ended questioning technique mentioned above to take the participant back to the point you’re interested in, and then just let them talk. Often they’ll take you step-by-step through what they were thinking. With the benefit of hindsight after completing the task, they may be able to give you better feedback than they could while they were only half way through it.
When the participant asks you a question
This is going to sound mean. If the participant asks you a question, answer with another question. For instance, they might say “What do you want me to enter here?” Your answer could be “What would you enter if you were at home/work?” That’s not a direction to enter certain information, but it allows you to learn both what the participant would normally do, and any reasons for their reluctance to enter it during the session (privacy or security concerns, etc.).
Obviously, if they are asking where the bathrooms are, it’s OK to give them a direct answer.