In a nutshell: Statistical significance testing compares two groups (e.g. people that play rugby vs. people that play football) and reports if they selected answers differently in a statistically robust way (e.g. way more football players selected "yes").

When your survey or your results are statistically significant, it means you can draw conclusions from your data with high confidence, as there is a low chance that the results of the survey came about by chance. This is especially useful if you want to test some hypothesis.

If you’re new to the world of consumer intelligence or if you're not entirely familiar with the statistical calculation models, don't worry, we are here to make life easy for you!

In order to run statistical significance test, you first need to have different groups for which you want to compare the results. Our results breakdown feature gives you the opportunity to easily split your results by demographic profiles or cross-reference the answers to multiple questions.

Step 1: Create your results breakdown

In the results dashboard, you'll find the Results breakdown page in the left-hand navigation bar. You can choose to breakdown your survey by demographic, question or segments. If you choose demographic or question, you'll be able to choose a single demographic or answer variable by which to break down all the questions in your survey. This will produce a matrix table for each survey question, with the variable as your columns and the answers to your survey questions as the rows.

Step 2: Choose your benchmark and confidence level

In order to run a statistical significance test it's important you choose a benchmark & confidence level.

The benchmark is the group you want to compare your results against. You can choose to compare your results against the total, this is the average of your survey, or against a specific group. In this example, if you want to see if there are any significant differences between Gen Z & Millennials, you can choose [18-25] as your benchmark.

The confidence level (or confidence interval) is the chance at which you can predict that if you would run the test again, you would get the same results. The confidence level is mostly set at 95%, but if you only need reasonable evidence and not "hard data" the 80% confidence level can be more than sufficient. For example when you are looking to get a general sentiment from customers or want to get a feel for market trends.

Step 3: Run the test!

With one simple click on the button you can see for all your questions (apart from open text questions) where there are significant differences between your groups and the benchmark group, at a given confidence level.

Significant results will be shown in bold. When the value is significantly higher than the benchmark, it will be blue with an upward arrow, when the value is significantly lower it will be black with a downward arrow.

Statistical significance is not yet included in the Excel export, but if you wish to export the data of a single question, you can do this by clicking on "Copy Results" and pasting the data into a spreadsheet, which will include the outcome of your statistical significance test.

Testing your data for significance is dependent on a number of factors. It's important that your survey is set up in an effective and bias-free way. This includes:

Also keep in mind that sample size can have a big impact on how likely you are to get statistically significant results. If this is important for you, and you need help choosing the right sample size, contact us via live chat, reach out to your Customer Success Manager or use one of our other channels.

Even if no answers are statistically significant, this does not mean that your survey has no value and that you cannot draw any conclusions from your results. How valuable statistical significance is can depend on the type of survey you are running and the type of data you want to gather.

For example, if you want in-depth feedback on a new creative, gathering a handful of real, qualitative responses could provide more than enough guidance. While not statistically significant, the data is still highly valuable.

Another case in which statistical significance might not be needed is when you’re looking for just a handful of insights to quickly sense-check an idea, or to confirm that you’re heading in the right direction.