Categories
design validation methods paper prototyping user testing

Paper prototype user testing

Pay no attention to the man behind the curtain. He’s working frantically to find the next sketch to show to the study participant. He might even be drawing it as we wait.  

Paper prototype usability testing is often called “Wizard of Oz” testing, and for good reason. Nothing is real, it’s all an illusion, and the illusion is controlled by people who are as unsure of the outcome as the test participant. But don’t worry, because that’s exactly how it should be.

Learn to love low fidelity

The less cost you sink in a prototype, the less concern you will have about throwing it out and starting again if you got things wrong.   The lowest possible cost for your prototype is to use nothing more than office supplies.  

two paper prototypes drawn on oversized paper with marker pens
Paper prototypes are fast to create and make the user testing process tactile and fun

Guerilla usability testing

One way mirrors in specialist usability labs are old school. They aren’t necessary for our purposes.

  • The fidelity of your test environment only has to match the fidelity of the code you are testing.
    • Because you are only testing paper, and your test is to verify your design by checking that the task flows rather than to emulate a real working environment, your test environment can be an office desk.
  • You don’t need any complex recording equipment for the user study. 
    • In fact, there’s probably not much point in recording it at all. You will be making changes to the UI almost immediately after the study, and you can write down any interesting observations on sticky notes.

There are other areas where you have to be just as professional as ever:

  • Make your participant feel welcome and comfortable.
    • Your participant is the most important person in the room. Offer them a drink when they arrive, help them understand what you’re up to, do everything you can to make them relax in this strange environment.
  • Be a good moderator
    • The best moderators say very little during a study session. Set the scene by telling the participant that the test is of the prototype, not of them. Explain that they are helping you make the software work as well as it can even at this early stage. Tell them they can stop at any time they want. Explain that you’ll be giving them tasks to perform and that you’d like them to think out loud as they go through the process. 
    • After the study starts, the only thing you should have to say is “Please remember to think out loud.” If the participant asks you a question about the UI, or where they should go next, ask them what they think they should do before jumping in to help them. 
  • Team members who are observing sit still and shut up.
    • Only the facilitator interacts with the participant. Everyone else in the room remains quiet and takes copious notes. They can ask questions when you’re done.

Users test the prototype, you don’t test users

User testing has nothing to do with testing users. It’s all about getting representative users of the system to help you test the match between your product and their expectations.

Your users don’t have to be very representative for these early prototype tests. We are testing whether the interaction ideas for the interface are likely to work. We don’t have real data or realistic setups.

  • The interaction ideas should be understandable by most experienced computer users, so that’s who you should recruit
  • If your interface contains specialist terminology or requires an understanding of certain procedures in order to use it properly, you’ll have to recruit more representative users. It’s still best to go with the lowest level of domain expertise that you can – you need to save the more skilled participants for later studies.
  • About 5 participants is enough to be sure that the problems you see are real

Use your scenarios to write tasks for users to perform.

  • Make sure that the wording of tasks doesn’t give away the answer. 
  • You will hand each task to the participant in turn, get them to read the task out loud (so that you know they understood it), and then watch as they perform the task. 
  • Asking participants to think out loud lets you know what they’re up to as they work.

One team member will play the role of computer. They are responsible for placing new UI elements on the table in front of the participant in response to where the participant “clicks” with their finger (or a pen).

  • Being the computer isn’t an easy job. It requires a good understanding of the intended path through the UI and the ability to improvise (normally with sticky notes) when the participant goes in a completely different but still very interesting direction than what you intended. 
  • It helps to have an hourglass card that you can put down over the UI while the computer finds and composes the next screen. That way the participant knows that you aren’t waiting for them to say or do anything.

Team members watch from a respectful distance. The best place is normally sitting behind the participant.

  • You can explain observers’ presence to the participant by saying that the observers are members of the team who are interested in finding out what areas of the product need improvement. There is no need to introduce observers by name or job title.
  • Each observer should be writing down user quotes and actions on sticky notes for later analysis. 
  • Observers do not interact with the participant during the session. We aren’t running a “formal” study, so talking to users won’t harm our data, but it’s just too overwhelming for the participant to have to deal with questions coming from so many strangers. Be respectful of the stress that participants feel, and give them just a single person to focus on during the session.
  • You might want to save time for a couple of questions from observers after the session finishes. Be sure you can trust your observers to ask polite questions.

Most paper prototype sessions are over pretty quickly, because it’s hard to create a large quantity of paper-based UI and then keep track of it all during the session.

  • Once you are done, find a way to thank your participants. Gift cards or marketing schwag normally go down well.

Leave a gap between each scheduled participant to review what happened and decide whether to make changes to the prototype.

  • Stick all the observations from each session up on the wall. Do this somewhere away from the testing environment, because some of the comments may be unintentionally disrespectful to the participants. Go through the stickies and group issues together. For subsequent sessions, just add to the existing groups.
  • If there is a clear and easy potential fix for an issue, and it’s likely that the issue is a problem with your interface rather than just a misunderstanding by the participant, make the change. 
  • If there is a clear and easy fix but you aren’t sure whether it was the interface or the participant’s understanding, discuss what you might do to fix it but wait to see at least one more participant before making the fix
  • If there is a tricky issue without a clear fix, note that it exists and prime everybody to watch for more clues with the next participant.
  • Save some time to re-arrange all the pieces of paper that the “computer” uses and to get the testing space ready for the next participant.

Make sure you document your final design decisions and any unresolved issues from the testing.

  • Film a quick video walkthrough of the final UI, or photograph/photocopy each stage in the process and turn it into a storyboard to pin on the wall.
  • Keep the sticky notes from the observations with the unresolved issues highlighted. If you made a storyboard, stick the notes on the relevant sections of the UI.

Check out a clip from a paper prototype testing session I ran.

A day well spent

You can easily run five participants and make all the resulting changes within one day. That day will pay for itself many times over in reduced rework and higher quality interfaces that users actually enjoy working with.

Paper prototype testing is a fun, educational activity that will make you run through several iterations of your design during the course of a day. The paper provides a physical interaction. Participants can pick bits of the UI up and move them, and they are much happier writing on the prototype to express their thoughts.

At the end of the test sessions you will have a better understanding of the tasks and of how to design UI to support them.

What’s next?

Now you can start coding!

  • The scenarios and paper prototypes can be easily deconstructed into user stories and user story maps to use for sprint planning.
  • You have already gone through all the churn and decision making for the UI as a team, so all the developers have a good understanding of what to build and why. 
  • If developers come across interaction issues during the development process, it’s likely that they’ll understand how users would react and so be able to build passable UI without further intervention.