Overcoming Cognitive Bias in User Research
Imagine you’ve just completed user testing around a new set of features and you’re feeling pretty good about how it went. You identified some serious problems, validated that users can complete a key task, learned something novel about how people work, and you’re ready to share your conclusions with stakeholders. Job well done!
Well, maybe… Researchers have identified more than 160 cognitive biases, many of which have the potential to distort your user research and lead you to faulty conclusions. Knowing something about these biases and following a few best practices can help researchers defend against them and come to more reliable findings.
At the UXPA Boston 2017 conference in late May, I attended a talk on this topic called “Best Check Yourself! Dealing With Cognitive Biases in User Research,” by Colin MacArthur. I’d like to share some of my takeaways from that talk plus some suggestions based on reading I’ve done since then.
What is cognitive bias and why does it matter?
You can think of cognitive biases as a set of mental shortcuts that help people make sense of the world. They help people move along with their lives without getting bogged down thinking about the meaning of every interaction or event. That matters for UX research because we do want to think about the meaning of every interaction or event, at least some of them, under certain circumstances, and mental shortcuts can easily lead us astray.
Following are several cognitive biases that stand out to me as being risky for UX researchers — afterwards, I’ll outline some steps you can take to contain their risks.
Recency, Availability, and Related Biases
Several biases describe the ways in which people assign disproportionate importance to things that stand out even when those things aren’t representative. The recency effect says we tend to remember and give greater weight to our most recent experiences; the availability heuristic says we make decisions by relying on information that comes most readily to mind; the peak end rule says we tend to judge an experience more on how we felt at its most intense point rather than on its average.
If I was doing user testing with ten users who successfully completed a task followed by two who struggled, the Recency Effect suggests I might overemphasize the significance of those final two users simply because they came last. If I interviewed a user who was particularly passionate — perhaps angry or upset — the Peak End Rule and Availability Heuristic say I might make more of their comments than was warranted simply because they were more memorable than others.
Confirmation Bias
We tend to give more weight to evidence that confirms our assumptions and beliefs and to discount data that doesn’t support those beliefs, which can affect user research in various ways. For example, I might tend to undervalue the significance of people being confused by a user interface widget if that widget feels completely natural or obvious to me.
Framing effect
According to the framing effect, we react differently to the same information depending on how it’s worded. On one level, it’s pretty obvious how to manage this bias: don’t ask leading questions. The thing is, it isn’t always easy to identify when language is leading. It’s plainly leading to ask, “isn’t this choice better?” But what about, “which choice do you prefer?”
In a classic study, researchers showed people two vacation destinations and said they had to make a choice between one or the other. Option A was described in bland terms, with average weather, average nightlife, and average beaches. Option B was described with greater positives but also greater negatives: lots of sunshine and gorgeous beaches but very cold water, very strong winds, and no nightlife.
Researchers asked one group of participants which vacation they would “cancel” and the other which they would “prefer.” The people who were asked which they would cancel were roughly even in selecting either option — but when asked which they would prefer, 67% chose option B. The descriptions didn’t change, but a mere word substitution dramatically affected which destination people would visit.
Selection Bias
This occurs when working with a group of study participants who haven’t been selected randomly, whether intentionally or not. The effect is that certain types of people might be more likely to be included than others, skewing results.
Anchoring Bias
This is the tendency to rely too much on the first piece of information when making decisions, and it’s used all the time in business negotiations. For example, if I was trying to sell you my car, I might say, “this is a pretty nice model — they usually go for $20,000 when they’re in such great condition!” You might want to pay less, but the number $20,000 is now in your thoughts, as is the notion that this car is in great condition. The negotiations are now anchored to these ideas in a way that gives me a real advantage.
Anchoring bias can play a role in user testing — for example, if you’re trying to see whether one workflow works better than another. Even if the first version is more complicated, a user may tend to prefer it simply because they saw it and figured it out first.
Clustering Illusion
This bias says that people tend see patterns even when there aren’t any, and it can be a particularly acute problem when working with small sample sizes, since two or three random events might lead us to think we’ve found a meaningful issue.
Empathy gap
People tend to underestimate how much their emotions affect their perceptions, attitudes, and behaviors. For example, if you’re angry because of a recent conflict with a colleague, the empathy gap suggests you may have trouble relating to someone who is feeling calm and collected. Similarly, if you’re feeling sad — even for reasons that aren’t related to work — you might be less receptive to the experience of a chipper and happy research participant.
Observer Expectancy Effect
A researcher’s biases can subconsciously influence study participants, according to the observer expectancy effect. For example, your point of view about how users will interact with a product can subtly inform how you ask a question and how a subject responds. This is difficult because you may not be aware that you have the bias to begin with, and — even if you are — you may not be aware if you’re conveying it.
Overcoming Cognitive Bias in User Research
For any cognitive bias, there are specific steps one can take to protect against its influence. That said, I think it’s best not to get hung-up on responding to each bias individually — there are so many, it tends to get a bit overwhelming. Instead, adhere to the following list of best practices for user research. They’re worthwhile guidelines in general, and they’ll also provide a practical and effective defense against cognitive bias.
- Create a research plan before you start. Know what questions you’re trying to answer and make decisions about how to run the study accordingly.
- Be thoughtful about how you screen and recruit participants, referring to your study’s objectives when deciding what kind of users you need to focus on. A group comprised mainly of college students might be perfect for research into a textbook trading app; it could be the cause of serious selection bias in research about something more general.
- Be explicit about your assumptions up front so that you’re aware of them and can consciously look for the opposite. Identify your assumptions with others on your team, list them out, and come back to them periodically to keep them in mind.
- Determine your process for analyzing results ahead of time and stick to it. For example, decide what standards of evidence your team will use for something to qualify as an actionable insight. If you see a certain response or behavior twice, what will you make of it? What if you see it five times? Thinking about this kind of thing ahead of time will tend to keep you honest in how you make sense of results.
- Make sure your sample size is large enough. Small sample sizes can be useful for certain things, but interpret your findings with circumspection. Deciding what size is appropriate isn’t obvious — the Nielsen Norman Group has a good article about this in terms of usability testing.
- Check your emotions. You can’t necessarily control your grumpy mood, but try to be aware of how you’re feeling when going into an interview — write it down and, when analyzing your findings, consider whether your mood could have had an effect.
- Consider evidence equally, paying close attention not to assign too much meaning to data that conforms to your preconceptions and assumptions. One way to do this is to consciously play devil’s advocate. When you see evidence that supports a hypothesis, make sure you’re also looking for evidence that disproves it.
- Write a script that you’ll follow when interviewing or observing users, and be careful to use open ended questions that don’t “lead the witness.” This is hard, so try to get feedback on your questions from colleagues, asking them to point out phrases that might inadvertently reveal your point of view.
- Think carefully about the sequence of questions, and consider alternating the order from interview to interview. This isn’t necessary for all questions, mainly ones that might trigger the anchoring effect — for example, if you were trying to understand whether users preferred one way of doing a task over another.
- Talk less, listen and observe instead. Ask a question and then get out of the way. Don’t rush to fill silences with your own observations or further questions.
- Watch your body language. Keep a poker face. Be neutral, careful not to respond positively or negatively in what you say or how you act. Obviously, this can be hard, but having a script can really help — it frees you up from thinking about your next question and lets you focus on keeping your tone and manner unbiased.
That may seem like a lot to keep in mind — it is a lot to keep in mind. So stick to the fundamental best practices and, when you have time, think about cognitive biases one at a time rather than all at once. To uncover ways in which your interpretations may reflect your own personal experience or biases, invite colleagues to observe your next testing session. Speak with them afterwards to see what they thought, and discuss whether and how your interpretations differ. Chip away at those cognitive biases bit by bit — and make your research findings as accurate and insightful as possible.
