Yes, There Are Bad Questions (Don't Ask!)

We all want answers. From tomorrow’s forecast to the winner of the next election, our news cycles, social searches, and conversations are full of questions.

With so much practice, then, why do so many people ask bad questions?

Bottom Line Up Front: It doesn’t have to be that way.

Bad questions, you say?

Yes, I know, I know. It’s not my aim to squelch curiosity or stifle learning. As a teacher, I’m reluctant to issue such a controversial statement, but there really is such a thing as a bad question — especially in survey design.

As a consultant and advisor to hundreds of survey designers over the years, I’ve witnessed plenty of — well, opportunities for improvement.

Where to begin?

Before we even dive down to the question level, it’s critical to be totally clear on two things: purpose and audience.

Why are you conducting this survey? If you don’t have a clear purpose, your survey will wander off-track and your participants won’t be along or the ride.
Who is your audience? Having a clear sense of your participants will help you to craft the most meaningful content and flow.

Once you’ve crafted your scope, you can take a closer look at your questions and identify any problems. Sometimes the issue is the question itself, sometimes it’s how the question is asked, sometimes it’s the question type used, and sometimes it’s something related — like the answer options provided or the order in which the question is presented.

Survey Question Improvement #1: Fix the question itself

Before you start asking your participant questions, start by asking yourself some questions. Once you’ve made a draft list, review it carefully. Remember that your participants probably don’t want to ask too many questions (if any!), they don’t want to be asked irrelevant questions, and they don’t want to get into any sensitive topics.

Before you ask them, ask yourself:

Why do you need or know the answer to this question?
If you don’t need to know, why do you want to know?
Is there any chance that you already have this information?
How will this person feel about being asked this question?
How will you plan to report on the results of this question?
How will you use the results?

If you can’t clearly answer these questions about your proposed query, cross it off and start again!

Survey Question Improvement #2: Ask the question in a better way

Even if you’ve figured out exactly what to ask, the way that you ask the question goes a long way toward improving clarity — both your participant’s understanding of what’s being asked and your understanding of how to interpret their responses.

In the rush to condense lots of questions into a few, for example, one of my favorite classics is the double-barrelled question — You like milk and cookies, right? No matter how your participant answers, you’ll never know if the like or dislike milk, cookies, or both.

Do any of the following seem familiar to you?

The Overexplained Question
- Q: When we (including but not limited to the company’s leaders and associates) move forward (in a legally binding sense) with the development of the new divisions (both geographical and topical), we are looking forward to seeing new leaders (at every level) step forward. Would you (personally, speaking on your own behalf) be interested in potentially entering into a leadership development program (with the understanding that interest does not indicate a commitment to enroll or a guarantee of acceptance, nor does acceptance in the aforementioned program guarantee attainment of a leadership position
- That feeling when you get to the end of a question and you’re not sure if a question mark is even warranted…

The Underexplained Question
- Q: Do you think Plan Z is good?
- That feeling when you have no idea what Plan Z is…

The Leading Question
- Q: With so many health benefits, wouldn’t you agree that a vegetarian diet is the best?
- How could anyone disagree?

The Misleading Question
- Q: You wouldn’t want to not miss out this opportunity, right?
- How could anyone understand?

The Logical Fallacy: Okay, there are just way too many of these to list, so let’s not get too Post Hoc, Ergo Propter Hoc here. Let’s just say that these are easy mistakes to make if you’re not careful. Want to brush up? Here’s a very cool reference on fallacies!

Survey Question Improvement #3: Use the right question type

Quick: What’s your favorite question type? If your preference shows through in your survey (Hello, Radio Button, Radio Button, Radio Button!) but the format doesn’t match your needs, you’ll be wasting everyone’s time.

Unless you’re in training (One of my favorite exercises!), you don’t need to include every question type in your survey. You do, however, need to use the right question type that best fits your question, possible answers, and reporting needs.

Most common survey question types: How well do you know your options?
Single- vs. Multi-select question type: How many answers should there really be?
Rating questions are the best: Should we just make everything a rating question?
Asking an open-ended question: Is the time spent by participants (writing the answers) and your team (reading and analyzing) worthwhile?

Survey Question Improvement #4: Improve everything else

There are so many more things that impact the success of a survey question, but let’s wrap up with just a short list.

Including the right answer options: Leaving one or more answers off of your list, adding too many answers, and mixing up participants with confusing phrasing or content that’s not parallel… yikes. Want to include different answer sets for different participants?
Ordering questions in a flow that supports your goals: Even if you’re not a professional researcher, the simple methodology of treating a survey as a conversation is one of my favorite approaches.
Design better, test often: Avoiding common survey design pitfalls isn’t impossible, but it’s a lot easier to avoid slip-ups if you have one or more people help you to test your survey. Test invitations can make a huge difference to ensuring that you’re hitting your project goals.

Ready to move forward? You’ve got this! A few references for your next steps:

Need some back-up? Book a support call from within your account or connect with your account manager. Need more help? Learn more about our Managed Services, including Survey Design Services, Survey Panel Services, and Fully Managed Projects.