Paper Prototype Usability Testing | August 30, 2003

I was up in London most of yesterday running a paper prototype usability test for an intranet project we're doing. If you're not sure what a paper prototype is, it's basically a paper version of a yet-to-be-built site. Paper prototypes can range from the very simplistic (often called low fidelity prototypes) to almost exact graphical representations of the site (often called high fidelity prototypes). For yesterdays test we were using fairly low fidelity prototypes which were little more than wireframes containing sample content.

Using paper prototypes to test a site is an extremely useful and practical thing to do. It's useful because it gives you a real insight into how people will use the final site, and always manages to throw up some really interesting and unexpected issues. If you're having a design dilemma, testing is often the best way to test your thinking. It's practical because paper prototypes are much quicker and easier to build than a real prototype, which means that testing can be done at a much earlier stage. This means you get user feedback much earlier in the process before you are too committed to a particular course of action.

Lo-fi prototypes are really good for testing general concepts early on in the design process. Because of their lack of design treatment, people tend to focus on bigger concepts like section naming, nav and user paths, rather than the look of the interface. Once the general concepts have been tested and refined, you can either test using hi-fi paper prototypes or possibly move straight to interactive prototypes.

Running a usability test using paper prototypes is actually very simple.You don't need a hi tech usability suite or anything flashy like that. Just a room with a desk, a notepad and possibly a video camera to record the users responses. Spend a bit of time planning the test. Work out what areas of the site you want to test and come up with suitable user tasks. Stick to a few core tasks 2-4 and aim for them to take between 30-45min in total. If possible, do a dry run with a friend/colleague before hand to make sure the timing works and you have everything you need.

Having a pre written script is a good idea. It should outline who you are, what you're doing and what you want from the test subject. You should explain that you are testing the the prototype not the user and encourage the user to talk aloud as much as possible. Explain that the moderator probably wont be able to answer questions during the test but there will be time afterwards to do this.

The moderators main job is to encourage the user to talk aloud and explain what they are doing. Use open questions like "what are you thinking" to encourage the user to think aloud. Often the user will ask question like "what does this button do" or "what should I do next". It's important not to prompt the user so you need to ask question back like "what do you think the button will do" or "If this were website what do you think you would do next". It's also a good idea to ask users why they did a certain action, what they expected to happen or what they understood by a certain term. This help clarify their thought processes much more.

The moderators job is also to take notes. However you tend only to notice the big things and it's often the small things that give away the users thought processes. As such I highly recommend using a video camera to record the sessions. For paper prototype tests I'll usually tape a folder or something to the desk and use this as the "computer screen", then focus the camera on this area. As long as the prototypes are on the folder you know that what the user does will be in shot. It sounds stupid but you must make sure you have enough tape for all the sessions and that you can plug your camera in and it will reach.

Being involved in testing is an extremely interesting experience. You'll be amazed when everybody understands things you thought would cause problems, and then don't get things which you thought would be obvious. You'll start to see that people don't actually read your carefully crafted headings and explanations. What they do is dive straight in and go for the first likely option they find. Only when they get stuck or make a wrong move do people actually start studying and reading things more. User testing is an amazingly enlightening process and one that more design teams should partake in.

Once the testing is over it's time to start the analysis. We create a table with the test subjects along the top and problems down the side. As we go thought the tapes, read our notes and look at the prototypes we will add problems down the side and then notes about how each user dealt with that problem. This way we can see if only one person experienced a problem of if it was more wide spread.

This information will be written up in a report which we give to the client, outlining the main issues and our recommendations on how to solve them. We'll use this information to amend the wireframes and inform the decisions we make in the design phase.

The whole process does take time. Usually around 3-5 person days. However it's well worth the effort. It allows us to test out concepts at a very early stage and prevents us from having to rework things latter on when change becomes much more costly. Testing gives us an amazing insight into how a site will actually be used and allows us to deliver a product that better meets user needs .Testing is also a lot of fun and being involved with usability tests can really help designers learn to build more intuitive and user friendly websites.

For more info on the subject of usability testing I'd highly recommend getting a copy of Don't Make Me Think! by Steve Krug.

Also worth a read is this usability testing pdf and presentation from Kelly Goto, author of Web Redesign Workflow That Works

Posted at August 30, 2003 4:48 PM


alannie said on September 3, 2003 5:43 AM

Great info! Thanks for taking the time to write this up. I’m wondering, what does a paper prototype actually look like? And how does it work - do users point to the link they want to “click” and then someone gives them the appropriate paper?

Andy Budd said on September 5, 2003 9:27 PM

We’ve created a bunch of standards widgets to use in Freehand. From elements, tab and the like. The level of detail you use in your paper prototype is really up to you. We tend to use very basic outlines (wireframes) of the pages with minimal design.

For paper prototype testing we usually have one person being the moderator and one person being the “computer”. The user will press links, fill in forms with a pen and press submit buttons on the paper like they would on screen and the “computer” then passes them the correct page.

It’s all actually very simple and quite fun. However there is quite a lot of prep work that needs doing. The building of the paper prototypes does take a while, but if you’re using wireframes you’re kind of killing two birds with one stone.