In a nutshell: Statistical significance testing compares two groups (e.g. people that play rugby vs. people that play football) and reports if they selected answers differently in a statistically robust way (e.g. way more football players selected "yes").

When your survey or your results are statistically significant, it means you can draw conclusions from your data with high confidence, as there is a low chance that the results of the survey came about by chance. This is especially useful if you want to test some hypothesis.

If you’re new to the world of consumer intelligence or if you're not entirely familiar with the statistical calculation models, don't worry, we are here to make life easy for you!

Everywhere you are comparing data, on the Analysis & Trends page, we are automatically calculating the statistical significant difference for you. On the Analysis page, you can easily split your results based on various segments, demographics or cross-reference the answers to multiple questions, and see where there are significant differences.

Go to Analysis on the results dashboard. On the right hand side panel, you can choose the variables that you want to add, either to your crosstab or charts. You can add demographics, dates, or answers to certain questions. You can also build your own segments by combining multiple variables together (e.g. Gen Z - living in London). By default, you will see the statistical differences highlighted in your crosstab or chart with an up or down arrow.

When running a statistical significance test, you can choose the confidence level as well as the column that you want to use as the comparison column.

The confidence level (or confidence interval) is the chance at which you can predict that if you would run the test again, you would get the same results. You can choose between 95%, 90% or 80%. The default setting is 95%, but if you only need reasonable evidence and not "hard data" the 80% confidence level can be more than sufficient. For example when you are looking to get a general sentiment from customers or want to get a feel for market trends.

The comparison column is the column you want to compare your variables against. You can choose to compare against the total column, the previous column or a specific column.

When you export your crosstab to Excel or copy the results, you will see the same upward & downward arrows, indicating which results are statistically significance. At the bottom of the question you will see what the columns are compared against and at which confidence level.

Testing your data for significance is dependent on a number of factors. It's important that your survey is set up in an effective and bias-free way. This includes:

Also keep in mind that sample size can have a big impact on how likely you are to get statistically significant results. If this is important for you, and you need help choosing the right sample size, contact us via live chat, reach out to your Customer Success Manager or use one of our other channels.

Even if no answers are statistically significant, this does not mean that your survey has no value and that you cannot draw any conclusions from your results. How valuable statistical significance is can depend on the type of survey you are running and the type of data you want to gather.

For example, if you want in-depth feedback on a new creative, gathering a handful of real, qualitative responses could provide more than enough guidance. While not statistically significant, the data is still highly valuable.

Another case in which statistical significance might not be needed is when you’re looking for just a handful of insights to quickly sense-check an idea, or to confirm that you’re heading in the right direction.