What is statistical significance?
In a nutshell: Statistical significance testing compares two groups (e.g. people that play rugby vs. people that play football) and reports if they selected answers differently in a statistically robust way (e.g. way more football players selected "yes").
When your survey or your results are statistically significant, it means you can draw conclusions from your data with high confidence, as there is a low chance that the results of the survey came about by chance. This is especially useful if you want to test some hypothesis.
How can I run statistical significance tests with Attest?
If you’re new to the world of consumer intelligence or if you're not entirely familiar with the statistical calculation models, don't worry, we are here to make life easy for you!
There are currently two ways to run statistical significance tests.
In your results breakdown you can run a significance test to see if there are any significant differences between your results. You can also use our beta statistical significance calculator in the results dashboard.
With our new beta statistical significance calculator, available for all your closed surveys, you can quickly check if your hypothesis has any significant results.
Go to the results page of any closed survey. You'll see the option "Sig. Test" in the right hand menu, next to "Filters" and "Targeting". Here you'll be able to check for significant results on your hypothesis.
Step 1: Select a target group
The first step is to select a "target group". This is the group of people of which you want to know if any of their responses are significantly different from another group. You create this group by choosing how they answered to a certain question (answer filters). You can add as many questions as you'd like, but we recommend choosing a maximum of two, otherwise your sample size might be a little too small.
Example: What sports do you play? - Football
Note: at the moment you can only use the statistical significance calculator to create groups with single and multiple choice questions.
Step 2: Select a comparison group
The second step is to select the "comparison group". This is the group you want to compare your target group with. You can choose "all respondents that do not fit the target group criteria", which will include all respondents from your survey apart from the ones that are in your target group, or create a specific group by again selecting the answer filters. Keep in mind that your comparison group has to have answered differently from your target group to at least one question.
Example: What sports do you play? - Rugby
Step 3: Select a question
Your third and last step is to choose your hypothesis question. This is the question for which you want to run the significance test and see if your target group has responded in a significantly different way. You do not need to select any answers; we will run tests for all answer options from that question.
Example: How likely are you to buy new clothing?
Check if you are happy with your settings. If you want to make changes to any of the steps, you can click "edit". Keep in mind that if you choose to change your target group, you will have to reselect your comparison group as well.
Once you are happy with your selections, you can click "run tests". The outcome of the test will let you know if there are any answers for which there is a significant different result between your target group and your comparison group, with a confidence level of 95%. This can be for all answer options, for no answer options or for a number of them.
Step 4: View your results
If you want to see how your groups have responded and which exact answers are significant or not, you can click on "view results" to see the table with the results for your two groups.
What's a good use case for this beta feature?
Check out some demo survey results HERE
Let’s say we’re a category manager for spirits and beer. If we wanted to advise retail outlets we distribute to which customer type (spirit drinkers v. beer drinkers) is more likely to purchase at their store, and we want high confidence in the recommendation, we could run a significance test to see if the difference between any of our results are significant at a 95% confidence level.
To use the calculator, we select Sig. Test in the top-left corner and we start our test. We select our first customer group we want to compare, which in this case is “spirits,” and we want to compare it to our beer drinking customers. We then pick the question related to which locations they prefer to purchase these drinks from and we run our test.
Turns out, some of our results are significantly different. When we click into the crosstab, we can see that spirit drinkers are significantly more likely to purchase alcohol at a liquor store vs beer drinkers. Conversely, beer drinkers are significantly more likely to purchase alcohol at a grocery or convenience store. We can therefore be confident that, were we to re-run this test, these results would remain the same. We can then use this information to persuade our liquor store owners and grocery store owners to target these customers in their displays.
How can I increase the chances of significant results?
Testing your data for significance is dependent on a number of factors. It's important that your survey is set up in an effective and bias-free way. This includes:
The order of your questions
Also keep in mind that sample size can have a big impact on how likely you are to get statistically significant results. If this is important for you, and you need help choosing the right sample size, contact us via live chat, reach out to your Customer Success Manager or use one of our other channels.
Do I always need statistically significant results?
Even if no answers are statistically significant, this does not mean that your survey has no value and that you cannot draw any conclusions from your results. How valuable statistical significance is can depend on the type of survey you are running and the type of data you want to gather.
For example, if you want in-depth feedback on a new creative, gathering a handful of real, qualitative responses could provide more than enough guidance. While not statistically significant, the data is still highly valuable.
Another case in which statistical significance might not be needed is when you’re looking for just a handful of insights to quickly sense-check an idea, or to confirm that you’re heading in the right direction.