In the UX space, when we gather data we rarely have access to the entire customer population and thus we rely on gathering data from samples of the customer population to make estimations of the population parameter. Usually, we estimate the sample means (i.e. arithmetic mean, geometric mean) and proportions (i.e. number of users that successfully completed a task). When we estimate a value from a sample, the estimation contains an error.
For example, if 8 out of 10 users completed a task successfully, you could estimate the task success proportion 80% (8/10), or if you would like to estimate how long it took users to successfully complete a task, you could estimate the arithmetic or geometric mean for the task times of the users who successfully completed this task.
The Confidence Interval (CI) contains the initial estimate (mean or proportion) plus or minus a margin of error and provides us with a range of values that we are fairly sure will include the true population value, such as a mean or proportion.
We say “fairly sure” to indicate that we have to choose how certain we would like to be in our assessment. This certainty is called the Confidence Level (CL) that we choose.
For example, in Table 1 the proportion of task success is 80% and the margin of error is 23.74%.
The 95% Adjusted Wald confidence interval falls between 47.94% and 95.41%. The lower confidence interval is 47.94%, while the upper confidence interval is 95.41%.
The CI indicates that the true task success for all customers (if we could measure it) is likely to be between 47.94% and 95.41%. |
The Confidence Level (CL) indicates how certain we would like to be in our assessment (It is the “95%” in statements such as “The 95% confidence interval falls between 47.94% and 95.41%”). Some common values for the CL are 95% & 90%, but other values could also be used (i.e. 99%, 85%, 80%) depending on how certain we would like to be in our assessment.
For example, in Table 1 the selected 95% CL indicates that if you were to sample 100 times from the same population and compute CI around the proportion of task success for each sample, 95 of those CI will contain the true population task success, while the other 5 will not. But you can not know if the produced interval contains the true population proportion (because 5 out of 100 times it will not, as we have selected 95% CL).
You could make the following statement: “We are 95% confident that the population task success rate is between 47.94% and 95.41%”.
The confidence interval tells us how much the parameter is likely to fluctuate, thus how precise the estimate is.
As the CI provides a range of values, the width of the CI shows how precise the estimate is. The width of the confidence interval equals two margins of error. It can also be calculated by finding the difference between the upper and the lower confidence interval. We review the factors that affect the width of the CI in another article.
For example in Table 1, the width of the CI is 47.47 percentage points wide. This can be found either by subtracting the upper CI from the lower CI (95.41% - 47.94%) or by multiplying the margin of error by 2 (23.74% * 2).