1. Scope of Experiments
What modules can merchants experiment on?
What modules can merchants experiment on?
In V1, merchants can experiment with:
Discounts
Payment O1ffers
Custom Checkout UI (excluding re-ordering of payment tiles and coupon re-ordering)
How many variants can a merchant create?
How many variants can a merchant create?
V1 supports two types of experiments:
Control + Variant A
Control + Variant A + Variant B
Note: The control group cannot be edited within the Experiments tab. To make changes, merchants must edit the standard checkout configuration.
How many experiments can be run at the same time?
How many experiments can be run at the same time?
Only one experiment can be live at a time.
This avoids overlapping experiments affecting the same target metric (e.g., conversions), which could make results inconclusive.
2. Experiment Setup & Management
Can an experiment be edited or stopped mid-way? Can timings be scheduled?
Can an experiment be edited or stopped mid-way? Can timings be scheduled?
Once live, variant configurations cannot be edited, to ensure accurate analytics.
Start and end times must be scheduled during setup.
An experiment can be stopped early, before its scheduled end time.
What kind of traffic splits are allowed?
What kind of traffic splits are allowed?
Merchants can split traffic in any percentage across variants, but the split cannot be changed once the experiment is live.
Can merchants preview each variant?
Can merchants preview each variant?
Yes. Variants can be previewed using the checkout link provided in the Experiments tab. Clicking the link will initiate checkout in that specific variant.
Can merchants create and save multiple draft experiments?
Can merchants create and save multiple draft experiments?
Yes. Up to 5 draft experiments can be saved. However, only one experiment can be live at any given time.
3. Analytics & Outcomes
How will merchants receive analytics for experiments?
How will merchants receive analytics for experiments?
Analytics are based on the primary metric selected as the goal during setup.
The winning variant is the one with the highest positive impact on the primary metric.
What outcome is expected after an experiment concludes?
What outcome is expected after an experiment concludes?
Experiment analytics provide a comparative analysis of each variant, based on the primary and secondary metrics defined at setup. Analytics are visible both during the experiment and after completion.
Can merchants view previous experiments and results?
Can merchants view previous experiments and results?
Yes. Merchants can access both:
Experiment configurations
Analytics results for past and current experiments