If you\u2019ve ever designed a survey, you\u2019ll know it\u2019s no walk in the park. Sure, it might be easy to build (if you have an amazing platform! ;)), but how can you be know you’re getting the questions right? And what other key considerations do you need to keep in mind? Turns out, designing a survey isn\u2019t just about framing the questions right, but about being aware of unintentional biases that often enter the process.<\/p>\n
This can often lead to you getting misleading results, or even answers that are completely off the mark. At best, it leads you to question the answers, at worst, you\u2019ll be making decisions based off inaccurate data that could cost your organization a pretty penny!<\/p>\n
So, here\u2019s a list of the biases you need to watch out for when designing your survey to help you capture data that is truly valuable.<\/p>\n
Where to begin? The best way to ensure that your survey stays on track and all the questions are relevant is by identifying your primary goal.<\/p>\n
Perhaps you want to understand the market appeal of a new product, or maybe you want to assess customer satisfaction. Whatever your goal is, it\u2019s imperative to keep it in mind to ensure you stay on track and ask only the questions that need asking. Too many questions and your respondents will likely face survey fatigue.<\/p>\n
When you\u2019re expecting a certain answer, it\u2019s easy to create questions that simply confirm your suspicions<\/a>. It could be in the form of not asking the right counter questions or even peppering in leading questions. (We should offer this in more colors, right?)<\/p>\n Remember to focus on your goal. Are you looking to understand how popular your new product will be, or are you looking to analyze the market perception? If it\u2019s the former, you might just end up creating a survey that confirms popularity and demand.<\/p>\n Let\u2019s fix that:<\/strong> <\/em>When setting the goal, run it by your colleagues or friends. An objective perspective can make all the difference at this point.<\/p>\n The questions you ask will determine the quality of the responses you get. It\u2019s also the stage where biases can really run wild, affecting the questions you write, the structuring of the survey, and ultimately the answers that roll in.<\/p>\n The way you phrase your question can often impact the way people would answer. For example, say you\u2019re just following up with guests at your restaurant. Just as they are about to leave, your survey asks, \u201cHow happy are you with the experience?\u201d<\/p>\n Even if they were neutral about the experience, the question leads them<\/a> to provide a more positive perspective.<\/p>\n Let\u2019s fix that:<\/strong> <\/em>Review your questions to ensure they are unbiased. In the previous scenario, asking respondents, \u201cPlease rate your experience here on a scale of 1 to 10\u201d or even using the NPS question and asking them, \u201cHow likely are you to recommend our restaurant to your friends and colleagues?\u201d will give you a more accurate idea of how your restaurant is perceived.<\/p>\n We often combine questions<\/a> without even thinking. For example, \u201cAre you happy and comfortable with your stay so far?\u201d But what happens when guests are unhappy bur still comfortable, or vice versa? That\u2019s where the inaccuracy really comes in. Even if someone answers “yes”, you can’t be sure whether they’re referring to one or both parts of the question.<\/p>\n Let\u2019s fix that:<\/strong><\/em> The easiest way to avoid this is to split the questions! While it\u2019s tempting to reduce the number of questions to avoid survey fatigue, it\u2019s arguably better to get reliable answers. So, the next time you find a conjunction in your question, do a double-take and consider making it two questions instead.<\/p>\n We often use absolute words like \u201calways\u201d and \u201cnever\u201d. But what happens when they are used in a survey? Asking them questions like, \u201cDo you always stay with us when you visit London?\u201d can bias the answers, because the answer is most likely, \u201cno\u201d.<\/p>\n Absolute language can also come off as aggressive, and make it difficult for respondents to answer with nuance.<\/p>\n In turn, respondents will often choose answers that don\u2019t perfectly align with what they\u2019d like to say, or simply abandon the survey.<\/p>\n Let\u2019s fix that:<\/strong> <\/em>Go through your survey once again and analyze the questions. Are you noticing any absolute language. Words like \u201call\u201d, \u201cnone\u201d, \u201conly\u201d, and \u201ceveryone\u201d are just a few more examples. If you notice these in your questions, consider replacing them to give respondents more space to maneuver. Consider updating to a question like “How often do you stay with us when you visit London?” and provide rating scale answers<\/a> that offer a range of frequency options.<\/p>\n Did you realize that the way you ask your questions or offer answer options can impact the way people respond? For example, if you ask guests what aspect about their experience frustrated them the most, followed immediately by how they would rate their experience with you, the answers might be skewed. That\u2019s because they\u2019ll recall their frustrations, thereby coloring their perception of the overall experience.<\/p>\n The same can be true for the way answers are ordered. Respondents who are asked about their favorite social media channels might often lean towards the first few answer choices, thereby impacting the reliability of the results!<\/p>\n Let\u2019s fix that:<\/strong><\/em> Randomizing is the answer! Randomize question order<\/a> and consider where and when to randomize the order of answer options, too! This helps you get answers you can bank on!<\/p>\n Everyone wants to look like their best self<\/a>. And this often translates to the way they answer your survey questions, especially when it comes to more sensitive topics.<\/p>\n For example, when asked how often people smoke, they might be more inclined to answer on the lesser side (or even never) depending on how the question is structured.<\/p>\n Let\u2019s fix that:<\/strong> <\/em>If your study focuses on smoking habits (or anything that might be sensitive to people), you need to ensure that the questions and answers are phrased in a non-judgmental way. Ask people something open and neutral like \u201cWhich of the following best describes your smoking habits?\u201d and provide simple, direct, answer options.<\/p>\n Data-driven decisions can help you grow exponentially. They remove guesswork from decisions, helping you forge forward in the right direction, that is, if the data is accurate.<\/p>\n Unfortunately, even our unconscious biases can show themselves in the survey and can misguide respondents into providing inaccurate data. This can shift the results, impacting not only the decisions we make, but the perception we build!<\/p>\n Of course, biases aren’t just an issue at the design stage of survey projects. Check out the next article on hidden biases that can impact your survey distribution and analytics!<\/p>\n Are you ready to build better surveys? Sogolytics<\/a> is the solution for you. Our innovative platform doesn\u2019t just guide you with the help of survey templates, but you can build a survey from scratch incorporating the best practices with the help of our Create with AI<\/a> feature.<\/p>\nCrafting questions<\/h2>\n
Bias Alert #2: Leading questions<\/h3>\n
Bias Alert #3: Double-barreled questions<\/h3>\n
Bias Alert #4: Absolute question bias<\/h3>\n
Bias Alert #5: Question order bias<\/h3>\n
Bias Alert #6: Social desirability bias<\/h3>\n
Unpacking biases in survey design: Why it matters<\/h2>\n