All Collections
Analysing results
Calculating Statistical significance
Calculating Statistical significance

Learn how you can check if your results are statistically significant or not

M
Written by Melanie Pieters
Updated over a week ago

What is statistical significance?

In a nutshell: Statistical significance testing compares two groups (e.g. people that play rugby vs. people that play football) and reports if they selected answers differently in a statistically robust way (e.g. way more football players selected "yes").

When your survey or your results are statistically significant, it means you can draw conclusions from your data with high confidence, as there is a low chance that the results of the survey came about by chance. This is especially useful if you want to test some hypothesis.

If you’re new to the world of consumer intelligence or if you're not entirely familiar with the statistical calculation models, don't worry, we are here to make life easy for you!

How can I run statistical significance tests with Attest?

In order to run statistical significance test, you first need to have different groups for which you want to compare the results. Our crosstabs feature gives you the opportunity to easily split your results by demographic profiles or cross-reference the answers to multiple questions.

Step 1: Add variables to you crosstabs

Go to crosstabs on the results dashboard. On the right hand side panel, you can choose the variables that you want to add as columns to your table, you can add demographics, dates, or answers to certain questions. You can also build your own segments by combining multiple variables together (e.g. Gen Z - living in London).

Step 2: Choose your statistical significance settings

When running a statistical significance test, you can choose the confidence level as well as the column that you want to use as the comparison column.

The confidence level (or confidence interval) is the chance at which you can predict that if you would run the test again, you would get the same results. You can choose between 95%, 90% or 80%. The default setting is 95%, but if you only need reasonable evidence and not "hard data" the 80% confidence level can be more than sufficient. For example when you are looking to get a general sentiment from customers or want to get a feel for market trends.

The comparison column is the column you want to compare your variables against. You can choose to compare against the total column, the previous column or a specific column.

  • Total is the default setting and is mostly useful when you are looking at multiple variables

  • Previous column is helpful is you create a crosstab based on dates and you want to see if there has been a significant increase or decrease vs. the previous date

  • Picking a specific column is helpful when you only have two variables in your crosstabs e.g. when comparing male .vs female or when you want to compare other brands vs. your own brand.

Step 3: Toggle on the test and dive into your results

With one simple click on the button you can see for all your questions where there are significant differences. If the results are significant higher vs. the comparison column they will be shown in green with an upward arrow, if they are significantly lower they’ll be blue with a downward arrow.

Exporting the data

When you export your crosstab to Excel or copy the results, you will see the same upward & downward arrows, indicating which results are statistically significance. At the bottom of the question you will see what the columns are compared against and at which confidence level.

How can I increase the chances of significant results?

Testing your data for significance is dependent on a number of factors. It's important that your survey is set up in an effective and bias-free way. This includes:

Also keep in mind that sample size can have a big impact on how likely you are to get statistically significant results. If this is important for you, and you need help choosing the right sample size, contact us via live chat, reach out to your Customer Success Manager or use one of our other channels.

Do I always need statistically significant results?

Even if no answers are statistically significant, this does not mean that your survey has no value and that you cannot draw any conclusions from your results. How valuable statistical significance is can depend on the type of survey you are running and the type of data you want to gather.

For example, if you want in-depth feedback on a new creative, gathering a handful of real, qualitative responses could provide more than enough guidance. While not statistically significant, the data is still highly valuable.

Another case in which statistical significance might not be needed is when you’re looking for just a handful of insights to quickly sense-check an idea, or to confirm that you’re heading in the right direction.

Did this answer your question?