After an experiment is ended, we can go ahead and run the analysis. We will ask for a dataset and column names, and KPI type to determine if we need to calculate IROAS or CPIC or CPA. The setting the objective is important to compute the iroas correctly in the case of holdout tests.
Channel CPIC at this stage is not necessary and will be removed.
The start date and end date of the experiment allows us to set the window we’re going to look at.
The budget determines how much budget we moved in order to ru the experiment and is essential to calculate the metrics precisely. In case of Lift tests it is how much budget they added. While in the case of Holdout Tests it is how much budget they took away compared to their average spend in this channel.
Lastly, we have to input the locations where the experiment took place.
Once all of that is done:
We will be presented with the significance and the significance level. The percentage change and actual when comparing the counterfactual and the actual results. We will also see these values represented in a waterfall graph, a cumulative graph to show how revenue dropped day by day since the treatment came into effect (a treatment start indicator will be useful).
We also have these plots below:
The first plot displays the counterfactual and the real values day by day. The second plot displays the difference between these two values day by day.