Are we building faster horses? Why surveys can't answer everything.
~ 10 Minutes to read
TL;DR: Surveys are a single tool in a vast research toolkit. Stick to basics and think through how you make them. Don’t use surveys to figure out what people want or what people do; use them to figure out how people feel about something. Surveys can’t help you understand why people act the way they do.
A long, long time ago, in a distant, far off magical place called Detroit, a man had an idea; make motorized carriages available to everyone. As Henry Ford started his little project, he did his homework. He investigated needs of people. The result? The Model T, and a giant automotive empire (and an extended family that continues to drive the Lions into the ground). But Henry Ford didn’t just ask people what they wanted. Why not, Henry?
“If I had asked people what they wanted, they would have said ‘faster horses’. Also I helped Barry Sanders retire.”
Thanks, Henry.
A survey isn’t always the right research method. And even when they are, surveys are really hard to do correctly. There are many pitfalls along the way.
If you stick to basics and are intentional about how you build them, surveys can be a valuable tool in your research toolkit.
Survey Basics
Ok great – we do surveys all the time. But what is your goal? What question are you trying to answer?
I want to uncover unknowns about what people do. (“What should I build?”)
I want to validate what we already know about what people do. (How is what I built performing?”)
I want to uncover unknowns about what people think and feel (“Do people like the system they have?”)
I want to validate whether the way people feel matches how I want them to feel (do people think what I built helps them?”)
All of these goals are worthy of research, but surveys cannot provide answers to all of them. Each goal has its own best practices when it comes to research:
“What should I build?” is a question that can be answered through observing people working in their natural context and studying actions in a controlled environment.
“How is what I built performing?” is often answered with analytics.
“Do people like the system they have?” is something that can be uncovered in focus groups and interviews.
“Do people think what I built helps them?” is where surveys really shine.
Surveys seem like they could help us with most of these goals, but human psychology and decades of user research have shown that if you need to know how people really use a system or what people really want, you need to go and observe it directly. If you need to know how someone feels, then you can use a survey.
This is why Henry Ford is often quoted as saying “If I had asked people what they wanted, they would have said faster horses”.
Unless your goal is to validate whether the way people think and feel meets expectations, you should not be using a survey.
Surveys as a research method
In research, all methods can be mapped on a set of axes of directness and what we seek to measure (behavior or attitude). Indirect methods are most useful for validating things we think we know; direct methods are best for discovering the unknowns in any project:
Adapted from: When Research Meets Reality, 2019 Midwest UX, Katie Briggs, NPR (@sisternebraska)
Why didn’t Henry Ford ask people directly? Because he hadn’t built anything yet. He didn’t know what he didn’t know. By not constraining himself to what people felt like they needed, he opened up all sorts of new possibilities.
In UX Research, we seek to do the same thing (except for continue the suffering of Lions fans. That can stop). We do a lot of “top-left” quadrant research in the form of usability studies and field studies. We measure behavior because we want to determine how successful a system will be in the hands of real people, not how successful a user thinks something will be. We measure directly because it’s about improving real people’s experience with systems, not some aggregate of what people do.
Why not just use a survey to ask people “is this system performing well”? There are well-documented tendencies (biases) for humans to not be aware of whether a system is truly useful. People will often sugarcoat their responses in surveys. Sometimes they will forget critical pieces of information. Remember that these email surveys are coming to a user regardless of whether they’ve recently used a system, so asking them to recall the last time they used it can be an effort in futility. If you want to know how a system works, you need to get closer to their use. You need to see them in their environment.
We aren’t trying to give them faster horses.
While often useful for evaluating sentiment on numerous topics, surveys are often a poor fit to actually get answers to questions related to the adoption of a system.
So what does a survey do?
Surveys cannot be used as a method of learning what to build or how to build it. They cannot be used to gather feature requests for future releases. But they are a great way to track how a lot of people think and feel about a system that has been created and is currently being used by end users. You can aggregate the information and get a great general understanding about a group of people. This information is a great way to spur more research in the other quadrants - good research begets more good research.
While we didn’t give them faster horses, we hear from a survey that people wish they could fuel their Model T with carrots. Carrots won’t be the solution, but we can translate their proposed solution into a new research question; what other ways could we fuel a Model T?
Surveys are best suited to learn how overall user groups feel about an implemented solution. From this sentiment, more research into why they feel like that can be started.
Survey Tools and Resources
We know we’re not trying to find faster horses now. So let’s find the best research method.
Maybe you’re thinking one of the other quadrants would be a better place to start your research. That’s great! UX research provides all sorts of solutions to answer any question you may have!
Or maybe you’ve decided we want to find out how people feel about what we’ve made, so we’re ready to do a survey. What’s your next step? Choosing a survey tool is a critical step, as is knowing how to design the survey correctly. I have experience in designing and creating effective, actionable surveys.
Good research is the first step in finding solutions past faster horses. I can help.
My wealth of background in research can be utilized for your benefit; contact me for a consultation.
Next up on the topic of Survey Design: Pitfall! How to make sure your survey doesn’t give you bad data.
Oh yeah…. One more thing.
Henry Ford never said that whole “faster horses” line... but hey, it’s a good saying, isn’t it?
“I’ve been misquoted for a century. Go Lions!”