Mastering Survey Design

Are you tired of struggling to design effective surveys that actually generate valuable insights? Many of us have faced this challenge while building products, where we know what we want to hear back but struggle to frame the questions in the most impactful way.

“We don’t prefer conducting surveys because it tends to give false positives and has a lot of leading questions” - VP of Product at one of the biggest banks in India.

“We tried surveys but users ask us to build something but don’t end up using it” - a YC founder with 7+ years of PM experience.

“I don’t get enough insights because I think I’m not asking the right questions” - Product Manager at a Series A startup.

With numerous options available, each question variant can elicit a slightly different response and create a distinct impression on the user. Additionally, cognitive biases must be considered to ensure reliable data. Amidst a busy workload, settling for lesser-than-ideal surveys becomes commonplace, resulting in predictable responses and a lack of new insights. It's a frustrating cycle of designing subpar surveys, obtaining unhelpful responses, and feeling like surveys are not worthwhile.

However, there are practical techniques to break free from this loop. The most successful product and research teams in the most product-driven companies actively use surveys to generate insights and validate their findings.

“.. in-app surveys provided answers to the two most pressing questions that kept me awake at night: why users spent so much time setting up [product], and why they stopped using it after only a few days.” - Senior Product Manager at fintech unicorn.

“In-app surveys make it easy to obtain timely, relevant responses. We've gained invaluable insights to guide our development process.” - Group Product Manager at a fitness unicorn.

“Insights from surveys are strong drivers of our product roadmap” - Cofounder and Head of Product at a Series B startup.

In particular, in-app surveys offer a powerful way to ask the right questions to the right users at the right time, resulting in highly accurate and actionable insights. In this article, we will delve into the top 10 cognitive biases (ranked from most common to least common) and explore how you can ask the "right" questions to overcome them. Let's dive in and unlock the full potential of your surveys.

1. Anchoring bias

This refers to the tendency of people to rely heavily on the first piece of information they receive (the "anchor") when making subsequent judgments or decisions.

Bad question: How much would you pay for this product?
Good question: What features are most important to you in a product like this?

Additionally, if you want to quantify, a question like “As you actively use [adjacent product], what value would you place on a product with these additional features? Please rate the value on a scale of 1 to 100, with 100 being the value of [adjacent product]?” is better.

Bad question: On a scale of 1 to 10, how satisfied are you with the app's performance? Good question: How would you describe your experience with the app's performance? Please provide specific details.

Tips on how to avoid:

1. Use open-ended questions to allow respondents to share their own opinions.
2. Randomize the order of response options to prevent the influence of anchor placement..
3. Use comparative questions that ask for comparisons between options to move the focus from a single anchor.

2. Availability bias

We (and your users) tend to judge the likelihood of an event by how easily examples come to mind. So if your users can easily think of examples of something happening, they are more likely to believe that it is likely to happen.

Bad question: How often do you use this feature?
Good question: What are the times when you would find this feature most useful?

Bad question: Have you experienced any technical issues with the app in the past week?
Good question: Have you experienced any technical issues with the app? If yes, please tell us about it.

Tips on how to avoid:

1. Instead of focusing on recent events, ask about experiences or observations over a longer period.
2. Include specific cues that encourage respondents to recall information from a variety of time frames or contexts.
3. Incorporate comparative questions that ask respondents to compare their experiences across different time frames or features.

3. Confirmation bias

We often look for information that confirms what we already believe, and we ignore or dismiss information that contradicts our beliefs.

Bad question: Do you like our new UI?
Good question: What do you like and dislike about our new UI?

Bad question: Do you think our product is easy to use?
Good question: What are some of the challenges you have faced when using our product?

Tips on how to avoid:

1. Use neutral language to avoid bias.
2. Encourage critical thinking and alternative viewpoints.

4. Curse of knowledge

We tend to overestimate how much our users know about something that we know a lot about.

Bad question: What do you think of our new feature?
Good question: Have you seen our new feature?

Bad question: How easy was it to use our app?
Good question: What was your experience using our app?

Tips on how to avoid:

1. Put yourself in the user's shoes. Try to think about what the user knows and doesn't know about your product or service.
2. Use plain language. Avoid using jargon or technical terms that the user may not understand.
3. Ask open-ended questions. This will allow the user to share their thoughts and feelings freely.

5. Halo effect

We tend to judge a thing based on one positive characteristic. If we like one thing about it, we are more likely to like other things about them as well.

Bad question: Do you think our product is the best on the market?
Good question: What are some of the things you like about our product?

Bad question: The app is now easier to use, how would you rate its overall functionality?
Good question: How would you rate the app's design? / How would you rate the app's functionality?

Tips on how to avoid:

1. Instead of combining attributes in a single question, ask respondents to evaluate each attribute separately.
2. Present respondents with a neutral set of product aspects.

6. Optimism bias

We tend to overestimate the likelihood of positive outcomes and underestimate the likelihood of negative outcomes. We may be more likely to take risks because we believe that things will work out in our favor.

Bad question: How likely are you to go to the gym in the future?
Good question: How often did you go in the last month?

Tips on how to avoid:

1. Avoid asking questions that ask the user to predict their future behavior or make plans for the future.
2. Focus on the user's current experience.
3. Use open-ended questions (you may be seeing a common pattern here

7. Representativeness bias

A tendency to make judgments or assumptions based on how closely something matches or represents a particular idealogy. It can lead to biased conclusions and judgments, overlooking individual variations and unique perspectives.

Bad question: Is our app the best in the market?
Good question: How does our app meet your specific needs and expectations?

Bad question: Are you satisfied with our app's performance compared to other apps?
Good question: What specific improvements would you like to see in our app?

Tips on how to avoid:

1. Ensure a diverse range of users are included in your survey, representing different demographics and user segments.
2. Focus on individual experiences - ask users to share specific details, challenges, and successes they have encountered while using your app, rather than making broad comparisons or assumptions.
3. Provide context and alternatives. For example, if you want to gather insights on users' experiences with other apps, offer a list of specific alternative apps and ask users to compare specific features or functionalities.

8. Self-serving bias

As product managers, we have a tendency to ask leading or biased questions that elicit positive feedback or reinforce preconceived notions about the app. This bias can result in skewed data and prevent you from identifying areas of improvement or understanding genuine user experiences.

Bad question: Did you find the app easy to navigate and intuitive?
Good question: What specific challenges did you face while navigating the app?

Bad question: How satisfied are you with the customer support provided by our app?
Good question: Please rate the quality of customer support provided by our app on a scale of 1-10.

Tips on how to avoid:

1. Use open-ended questions to allow users to share unbiased feedback and highlight both positive and negative aspects.
2. Request users to provide specific examples based on their actual experiences rather than generalizations or assumptions.

9. Social desirability bias

We tend to answer questions in a way that makes us look good. We may avoid answering questions honestly if we think that our answers will make us look bad.

Bad question: Do you think our product is easy to use?
Good question: What are some of the challenges you have faced when using our product?

Bad question: Do you think feature A meets all your needs?
Good question: What aspects of feature A do you think could be improved?

Tips on how to avoid:

1. Avoid using overly positive or leading language that could influence users to provide socially desirable responses.
2. Emphasize that responses are confidential to alleviate concerns about judgment.
3. Separate questions that combine multiple concepts to ensure each aspect is addressed individually (double-barreled questions).

10. Sunk cost fallacy

This fallacy can create bias by leading respondents to give inaccurate or biased answers in order to justify their past decisions.

Bad question: What would it take for you to give up on our app?
Good question: What do you dislike about our app?

Tips on how to avoid:

1. Avoid asking questions that focus on the user's past investment
2. Use open-ended questions that allow users to share their thoughts and feelings freely.

Fixing these cognitive biases in your survey question design will empower you to extract genuine and valuable insights from your users. Get ready to unlock a wealth of knowledge and propel your product to new heights!

‍