Skip to main content
Avoiding bias in your survey

How to write your survey to avoid different types of bias that can skew your results

Updated over a week ago

While we aim to provide the best quality sample, your survey results can still contain inaccuracies if the survey is written in a way to encourage dishonesty. Oftentimes your respondents aren't aiming to give you incorrect responses, but the question order, wording and settings can all subconsciously steer your respondent to give an answer which isn't quite true. This is called biasing your respondents, and below we detail the different types of bias and how to avoid each one.

For more on avoiding bias, and other top survey-writing tips, head over to our Consumer Research Academy.

Agreement bias

Also known as acquiescence bias, agreement bias occurs when you present your respondents with a simple yes/no question. As humans we're all more likely to agree than disagree with a statement posed by a stranger. As such, using yes/no answer options can lead to overclaiming for the "yes" answer.

It's very easy to solve this bias. The question just needs to be rephrased to remove the positive and negative sentiments, so the answer options are neutral and it's not clear to the respondent which they need to select to progress through the survey.

For instance:

Can be rephrased to avoid agreement bias as:

For more information on how to write surveys to avoid bias, don't hesitate to get in touch with our team using the in-platform live chat.

Order bias

Order bias can affect both questions and answers. Taking each in turn:

Order bias in questions

Order bias occurs when asking a certain question - or presenting a certain media item - first has an impact on the frame of mind or knowledge of the respondent as they go into follow up questions. For instance, if presenting a series of adverts, all subsequent adverts will be compared to the earlier ones in the respondent's mind, even if not consciously.

There are two ways to mitigate order bias in questions:

  1. Make sure you're not giving away answers to later questions with earlier ones. Consider the order in which you ask questions to make sure you move from the most general at the start to the most specific towards the end of the survey.

  2. Use the Randomise questions button in the left hand card list of the survey editor to shuffle similar questions, or groups of questions & text cards. This means each respondent will see those randomised questions in a different order, mitigating order bias in your survey.

Order bias in answers

Similarly, respondents are more likely to select answer options at the top of the list than those towards the bottom. If left unrandomised this could mean you see overclaiming on the answers at the top of your single or multiple choice answer list.

Removing order bias in answers is very easy. Unless you're using a scale, you should always toggle the Randomise answers button on, meaning the answer order will be shuffled for each respondent. You can then Pin certain answers in place if you just want to shuffle the other answer options around them.

Leading Questions

A leading question is one that is framed in such a way that it encourages respondents to give a particular answer. They should be avoided in surveys to avoid unfairly influencing the results.

For example: ‘How uncomfortable do you find outdoor furniture?’ This question presupposes that respondents find outdoor furniture uncomfortable in the first place. It would be better worded as ‘How would you describe your comfort level when using outdoor furniture?’

If you have any questions around bias, please feel free to reach out to your Customer Research Manager, or contact us via our live chat in the bottom right of this page.

Did this answer your question?