Quantitative vs. Qualitative Research: Which Should Come First?

Every few weeks, our team has a high-level chat about a UX topic we’re interested in at the moment. Someone poses a question on our DIG (design interest group) thread—usually about a recent challenge or something they’re learning about—and the rest of us weigh in with thoughts and questions of our own. This is a great practice for the team because it allows us to expand our collective knowledge on a regular basis.

Alli started our most recent chat about qualitative and quantitative research. When should we use one approach over the other? When should we use both? And in what order?

We really enjoyed the conversation that followed, and we thought you would, too. Here’s a transcript of the chat. Feel free to join the conversation and share your own thoughts or questions in the comments!

* * *

Allison Hossack

Shawn recently posed an interesting question: “what comes first, quantitative or qualitative research?” Like the chicken or the egg, I think it depends on who you ask.

I usually conduct quantitative first (often in the form of a survey), followed by qualitative. What I like about conducting a survey first is the ability to ask users if they’d be interested in a follow-up conversation so that we can probe deeper into their answers. I also love clustering common survey answers to help inform persona creation.

However, I’ve been listening to podcasts and reading about the Jobs to be Done methodology, and now I can also see the value in doing quantitative after qualitative research. You create some hypotheses based on talking with real users, then validate those ideas across a larger sample.

I love numbers, and there is power in them. My answer changes based on my experience and the context of a project. This Dollars to Donuts podcast discusses different research disciplines and says the answer depends on the question being asked.

Survey questions can bias users if not worded very carefully. Also, some users will tell you what they think you want to hear. But what I like about surveys is the ability to consistently ask the same questions across many users. Surveys level the playing field by adding common constraints and methods. Assuming all users are honest, a survey can provide much more objective data than our interpretation of what we think they feel or really need. Here is a perspective Shawn found from Intercom.

What do think? Quantitative first? Qualitative first? Which do you prefer?

Phil Charron

Wow, what a great question. My opinion is very similar and can be summed up the same way we sum everything up: “It depends.”

When I have a limited budget or only one practical choice, I tend to default to qualitative. I feel the human connections we make and the ability to go down undiscovered pathways far outweighs the volume of responses a survey can offer.

I am a huge fan of the methods we’ve developed with other clients in the past: quantitative so we can do cluster analyses, then qualitative so we can drill down into the why of those clusters.

I get the strengths of the opposite approach, though. Validate what you hear in qualitative with large-scale, focused surveys. Don’t discount the value of using analytics and other behavioral data sources for that validation as well.

My ultimate preference: ongoing research, either driven by design or at least with heavy design input/participation. An established research budget that isn’t tied to specific research methods is a design team’s ultimate toolbox. It may be more valuable than (gasp) Sketch.

Sunkwon Bush

It’s been interesting to see that for some past projects, starting with quantitative research had limited value, but for others, leading with qualitative had its own set of issues.

One thing’s for sure, though: our clients’ awareness of their ability to create bias and their willingness to take the steps to prevent it is key for a successful research project, no matter what type of research.

I would love to be able to always start with very early qualitative interviews to help shape my focus for a phase of quantitative survey, then, based on those survey results, conduct more focused interviews again. Having that ongoing flexible research budget would be amazing because it would make it easier to change course and alter our approach based on what type of research is most effective.

Brittany Murphy

Love this topic, Alli!

I’m all about qualitative research. In a perfect research world I’d say I would split my efforts 95/5 for qualitative/quantitative. Disclaimer: we focused mainly on qualitative research methods during grad school and didn’t touch much on quantitative.

To me, the area where quantitative wins over qualitative is speed. It’s great for a starting point or at times when your explorations or thoughts are diverging and you’re unsure what to dig into first. I’ve also been seeing some cool tricks Jesse has been pulling with Survey Monkey results on trends and relationships of different user groups.

I also believe real users with real lives, real quotes, and real use cases provide a much richer and more powerful story than numbers ever can.

Allison Hossack

@Phil – your point about ongoing research reminded me of some nice tools I’ve noticed that allow data collection right at the point of context.

@Brittany – speaking of “real quotes,” one of my favorite parts of our quantitative research is reading and incorporating answers from open-ended text questions. I find these to be super powerful.

Phil Charron

I hear what you’re putting down, Brittany, and I’m often in that camp.

I think the times that we’ve leaned over to quantitative, especially to start, we had no context for the behaviors of a large mass of people in an audience with very diverse needs, goals, and priorities.

In previous projects, we’ve learned a lot looking at these groups as a mass first for the what, then drilling down into the why with qualitative work. Qualitative was SO important to finish those stories.

Jesse Livingston

Awesome discussion, everyone. I’m loving this topic.

I would never say that one approach is “better” than the other. To me, that’s like saying a spoon is better than a fork. Neither are intrinsically “better,” they just serve different purposes. So the question becomes not whether a spoon is better than a fork, but are you eating a kale salad, or gorging yourself on a giant bowl of Captain Crunch (@Shawn)?

I would highly recommend looking at Tomer Sharon’s thoughts on the four different research methods in his book “Validating Product Ideas.” In it, he defines what kind of question each research method excels at. Again, it’s not a question of “better or worse”, it’s a question of “which is the right tool for the job?”

Quantitative: We’ve already touched on this, but I think quantitative answers the question, “What are people doing?” The reason quantitative is generally a better gauge of actual behavior than qualitative is because, well, people lie. Not maliciously of course—we just aren’t always very good at describing our behavior.

Qualitative: But of course, the key is understanding “Why are they doing X?” which is where qualitative excels. Like others, I love opening with qualitative because it’s a great exploratory approach that allows you to create assumptions to later validate or invalidate across a larger population.

Other methods: A few methods that can combine the best of both worlds of course, which we don’t get to do as often, are diary studies and field observation/contextual inquiry. Both allow for human interaction, but there’s also a measure of objectivity.

Brittany Murphy

Phil: I totally get what you’re saying. I understand the value of true quantitative methods and I’d love to learn more about how to get the actual what out of that. My (limited) experiences with quantitative have provided questionably believable results, which is why I always use it as a starting point to jump into qualitative and verify everything with field observation and interviews.

Jesse mentioned diary studies. I have experience with that method in the past relating to a project focused on diabetes. It’s almost like getting field observation without being there. There are many tools out there (dscout is one example) that we may be able to explore.

Phil Charron

Brittany and Jesse: I’d love to learn more about diary studies—seems like a great alternative to the “I want to please you” or “I want to argue with you” effect of interviews.

Sunkwon Bush

Diary studies would be great as part of an in-studio DIG focused on research methods!

Shawn Hickman

Loving this discussion. Learning a lot too! 🙂

Russ Starke

This is awesome, y’all. Some quick thoughts:

I would argue that in most cases we do a hybrid of qualitative/quantitative/qualitative inherently, since we can’t create the quantitative survey without first having some qualitative discussions with clients. Sure, in that case they’re biased and giving you their own thoughts on the audience, but then you get to test those things out in qualitative and see if they’re true for a statistically relevant amount of folks in the target audience.

But to echo both Phil and Brittany—there are times when quantitative does and doesn’t make a lot of sense.

When it does: if you’re helping a client determine if a product or service is really likely to fly for a significant amount of the consumer population or some other more narrowly defined external audience that they are trying to engage. If they really want statistical confidence of the unmet want/need, it’s not going to come from having 10 conversations, no matter how great they are.

When it doesn’t: if you’ve got a pretty well-known target audience and a throbbing unmet need (that, if you address it, will be a huge win no matter what), then it can absolutely be an exercise that doesn’t net much new knowledge. Can it still confirm assumptions across a broader set of users? Sure, but after the fact you can feel a bit like “eh, not sure what we really go out of that.”

As with most things, there’s usually a mix of science and alchemy at play.

Abby DePrimo

I love everything about this thread. Thank you all for being such smartypants—I feel like I’ve learned more about this topic from this exchange than poking through books and articles the past few weeks.

Phil Charron

Brad Sukala

I see the value of using qualitative to ferret out answers to our unknown issues early on in process. Based on my understanding, this is exactly why JTBD methodology follows a qualitative>quantitative approach. I think Phil made the point of using an early quantitative to find the what followed by the why. I think JTBD would probably replace the why with how.

Following up on Phil and Brittany’s observations that quantitative is potentially less valuable for our kind of work, I can’t argue with that. It is very much a function of the questions at hand, though. If it purely is an exploration of “the same things, but differently,” then I agree that qualitative is likely sufficient. As soon as we start veering into the territory of “fix it, it’s not working,” then we likely have to be more willing to mix quantitative and qualitative. Quant is never going to tell us exactly what people truly want, but it’ll help us define the edges, just like scientists use the wobble of distant stars to ascertain the presence of planets they cannot see.

Brittany Murphy

@Brad: I’m not sure I agree qualitative is only good for situations with “the same things, but differently.” I think qualitative works better to expose areas of unmet needs that were never even considered and open up a different path for design, specifically through field observation.

This article explains my view of qualitative research well—they use the term “radical or disruptive innovation” to describe the outcome from this form of research.

“For incremental innovation, by definition, there is a history of actual market performance against which to calibrate new concepts, so it makes sense that we assess ideas using processes and objective pass/fail criteria that have proven to be good predictors in the past. Unfortunately, these same processes often work against our ability to innovate effectively in more radical ways – to create ‘disruptive’ innovations.”

Brad Sukala

@Brittany: You’re right. I didn’t write that very clearly. I meant that we could probably get away with only qualitative if it’s a simple matter of “the same but different.” Ideally we’d like to do more, but that would get us to a minimal baseline. I am 100% on board with that quote.