The way to create good study questions is to work out what you want to be able to say before you even think about the methods you’ll use to find the answer.
Formulate your answer statement
Write down the statement that you’d like to be able to make. This helps you work out how you’re going to get to the answer.
- “80% of users preferred option A over option B”
Find out what kind of study will answer your question
Based on the answer statement you want to make, use the cheat sheet to determine what kinds of studies you could run.
- In this instance, we are looking to compare things, so either an A/B test or a survey could be used. Because we are asking for preference, we’ll use the survey method.
Make sure your statement is measurable
What does it mean to “prefer” something? We could show both options and ask, but then we don’t see how much one option was preferred. Instead, let’s ask for a rating for option A and a rating for option B and then compare those ratings. Ensure that your comparison scale is one that can be measured. A good measure might be satisfaction with the outcome, or confidence that the task was successful.
- “80% of users were more satisfied with option A than option B”
Ask yourself how much of a difference matters
For instance, users might show a strong preference or might not really care.
- “80% of users were more satisfied with option A than option B by at least 2 points on a 5 point scale”
Ask yourself what could introduce bias
For instance, if everyone saw option A first, it might prejudice them.
- “80% of users were more satisfied with option A than option B in an randomized presentation by at least 2 points on a 5 point scale”
Formulate your research question
Write down your question in a way that you think it could be measured.
- “Users will be asked to perform a task with both option A and option B. Presentation order will be randomized. After performing the task, users will mark their satisfaction with each option on a five-point scale ranging from very dissatisfied to very satisfied.”
By working backwards from the desired answer to the question, you can be sure that you chose the right research methods and that you get valuable results.