If you’ve ever designed a survey, you’ll know it’s no walk in the park. Sure, it might be easy to build (if you have an amazing platform! ;)), but how can you be know you’re getting the questions right? And what other key considerations do you need to keep in mind? Turns out, designing a survey isn’t just about framing the questions right, but about being aware of unintentional biases that often enter the process.

This can often lead to you getting misleading results, or even answers that are completely off the mark. At best, it leads you to question the answers, at worst, you’ll be making decisions based off inaccurate data that could cost your organization a pretty penny!

So, here’s a list of the biases you need to watch out for when designing your survey to help you capture data that is truly valuable.

Setting the goal

Where to begin? The best way to ensure that your survey stays on track and all the questions are relevant is by identifying your primary goal.

Perhaps you want to understand the market appeal of a new product, or maybe you want to assess customer satisfaction. Whatever your goal is, it’s imperative to keep it in mind to ensure you stay on track and ask only the questions that need asking. Too many questions and your respondents will likely face survey fatigue.

Bias Alert #1: Confirmation bias

When you’re expecting a certain answer, it’s easy to create questions that simply confirm your suspicions. It could be in the form of not asking the right counter questions or even peppering in leading questions. (We should offer this in more colors, right?)

Remember to focus on your goal. Are you looking to understand how popular your new product will be, or are you looking to analyze the market perception? If it’s the former, you might just end up creating a survey that confirms popularity and demand.

Let’s fix that: When setting the goal, run it by your colleagues or friends. An objective perspective can make all the difference at this point.

Crafting questions

The questions you ask will determine the quality of the responses you get. It’s also the stage where biases can really run wild, affecting the questions you write, the structuring of the survey, and ultimately the answers that roll in.

Bias Alert #2: Leading questions

The way you phrase your question can often impact the way people would answer. For example, say you’re just following up with guests at your restaurant. Just as they are about to leave, your survey asks, “How happy are you with the experience?”

Even if they were neutral about the experience, the question leads them to provide a more positive perspective.

Let’s fix that: Review your questions to ensure they are unbiased. In the previous scenario, asking respondents, “Please rate your experience here on a scale of 1 to 10” or even using the NPS question and asking them, “How likely are you to recommend our restaurant to your friends and colleagues?” will give you a more accurate idea of how your restaurant is perceived.

Bias Alert #3: Double-barreled questions

We often combine questions without even thinking. For example, “Are you happy and comfortable with your stay so far?” But what happens when guests are unhappy bur still comfortable, or vice versa? That’s where the inaccuracy really comes in. Even if someone answers “yes”, you can’t be sure whether they’re referring to one or both parts of the question.

Let’s fix that: The easiest way to avoid this is to split the questions! While it’s tempting to reduce the number of questions to avoid survey fatigue, it’s arguably better to get reliable answers. So, the next time you find a conjunction in your question, do a double-take and consider making it two questions instead.

Bias Alert #4: Absolute question bias

We often use absolute words like “always” and “never”. But what happens when they are used in a survey? Asking them questions like, “Do you always stay with us when you visit London?” can bias the answers, because the answer is most likely, “no”.

Absolute language can also come off as aggressive, and make it difficult for respondents to answer with nuance.

In turn, respondents will often choose answers that don’t perfectly align with what they’d like to say, or simply abandon the survey.

Let’s fix that: Go through your survey once again and analyze the questions. Are you noticing any absolute language. Words like “all”, “none”, “only”, and “everyone” are just a few more examples. If you notice these in your questions, consider replacing them to give respondents more space to maneuver. Consider updating to a question like “How often do you stay with us when you visit London?” and provide rating scale answers that offer a range of frequency options.

Bias Alert #5: Question order bias

Did you realize that the way you ask your questions or offer answer options can impact the way people respond? For example, if you ask guests what aspect about their experience frustrated them the most, followed immediately by how they would rate their experience with you, the answers might be skewed. That’s because they’ll recall their frustrations, thereby coloring their perception of the overall experience.

The same can be true for the way answers are ordered. Respondents who are asked about their favorite social media channels might often lean towards the first few answer choices, thereby impacting the reliability of the results!

Let’s fix that: Randomizing is the answer! Randomize question order and consider where and when to randomize the order of answer options, too! This helps you get answers you can bank on!

Bias Alert #6: Social desirability bias

Everyone wants to look like their best self. And this often translates to the way they answer your survey questions, especially when it comes to more sensitive topics.

For example, when asked how often people smoke, they might be more inclined to answer on the lesser side (or even never) depending on how the question is structured.

Let’s fix that: If your study focuses on smoking habits (or anything that might be sensitive to people), you need to ensure that the questions and answers are phrased in a non-judgmental way. Ask people something open and neutral like “Which of the following best describes your smoking habits?” and provide simple, direct, answer options.

Unpacking biases in survey design: Why it matters

Data-driven decisions can help you grow exponentially. They remove guesswork from decisions, helping you forge forward in the right direction, that is, if the data is accurate.

Unfortunately, even our unconscious biases can show themselves in the survey and can misguide respondents into providing inaccurate data. This can shift the results, impacting not only the decisions we make, but the perception we build!

Of course, biases aren’t just an issue at the design stage of survey projects. Check out the next article on hidden biases that can impact your survey distribution and analytics!

Are you ready to build better surveys? Sogolytics is the solution for you. Our innovative platform doesn’t just guide you with the help of survey templates, but you can build a survey from scratch incorporating the best practices with the help of our Create with AI feature.

Start with a free trial today and discover the Sogolytics difference.