What is the "A/B tests" option for?
A/B testing, also known as split testing, is a method to compare two versions of an onboarding. The purpose of A/B testing is to determine which version performs better by presenting different variations to different segments of your audience and measuring their response.
Does it function for both iOS and web2app?
Currently, AB split testing feature functions for iOS ask only. Web2App doesn't support AB testing but our team working on it.
What can I expect to see in the main "A/B tests" tab?
On the dashboard of the "A/B tests" tab, you will see the option "+ ADD TEST" and the panel with the columns "Name," "Current env," "Variants," and "Actions."
How to add an onboarding to the "A/B test?"
Create a project and an onboarding.
Go to the AB tests tab and click "+ADD TEST" and you will see a pop-up:
In the pop-up, enter the A/B test name and select onboardings for your groups.
You can add more groups by clicking "+Add," delete groups by clicking "Delete," and adjust thresholds using sliders.
To finalize the A/B test, click "Done" to save it on DEV or "Release to QA" to save and release it for QA.
Note: You can add/remove groups and select onboardings only when onboarding is not published to QA/Prod.
The new test will appear with the A/B test name, environment, variants, and actions visible. If you add more tests, the latest one created will show first.
How many A/B tests can I add?
In the DEV environment, you can add as many tests as you need for your app without any limitations. However, only one A/B test can be released at a time in the QA or Prod environments.
Is entering a name obligatory for A/B tests?
Yes, indeed. You will have to enter a valid title for your test so that you can differentiate and monitor the progress and effectiveness of your current onboarding being tested. The name field accepts special symbols, emojis, numbers, and spaces, and also text in different languages.
How to add the needed onboarding to Groups B,C, D... in A/B tests?
Onboarding distribution to a certain group in A/B tests is an automatic process once you add more variants by the "+ADD" button.
There you should choose the needed onboardings and you can change the percentage of how often it will be displayed to this particular group. Moreover, you may opt for the same onboarding to be displayed to different groups depending on the Threshold percentage stated by you within the current test.
By default, group titles are labeled as Group A, Group B, and so on until Group Z. If you add more than 35 variants, additional groups are labeled with a # sign, for example, Group A #, Group B #, and so on.
Can I add unpublished onboardings to an A/B test?
Yes; if you have onboardings that are not published to QA yet, you can still see them on the drop-down menu and select them for your A/B test. You can then create a test with several onboardings needed.
Can I edit the onboarding that has already been added to an A/B test?
Yes; you may apply whatever alterations only to the unpublished onboarding in the DEV environment that has already been added to an A/B test and modify the required settings by clicking on the needed group right on the main dashboard of the "A/B tests" module.
However, please note:
You can't edit onboarding from the AB test if your AB test is released to QA or PROD.
If the onboarding has been added to the A/B test cannot be published separately to QA or PROD.
Can I modify the already published onboarding added to an A/B test?
No; in such a case you should clone the onboarding and apply the needed corrections in the settings module. The system is designed to limit unexpected and unsolicited changes by other project members in the already published onboarding.
What happens if an onboarding in the dev environment is already in an A/B test?
If an onboarding in the dev environment is already in any A/B test, you will not be able to add it to another A/B test.
What is the "Threshold" slider in A/B tests for?
It helps you allocate the percentage among the onboardings and groups being currently tested. You need to select onboardings and set the needed thresholds, for example, if 50/50 is selected, it means that 50% of your actual customers will see Group A while the rest 50% of current active users will see Group B.
Can I change "Threshold" when A/B tests on QA or Prod?
Yes, indeed, can change the threshold any time when A/B tests on Dev, on QA, and Prod
What environments are available for added A/B tests? What can I do with the tested onboardings?
You can test onboardings in DEV for ongoing adjustments, QA for testing the set onboarding, and PROD for the final version shown to customers without further changes.
What happens when I publish 2 A/B tests to QA?
Let's break it down in a simpler way:
Imagine we have two tests, A/B test 1 with 2 groups (Group A has Onboarding 1, Group 2 has Onboarding2) and A/B test 2 with 2 groups (Group A has Onboarding 3, Group 2 has Onboarding4). We release A/B test 1 to the QA, and later, we release A/B test 2 to the QA also.
After releasing A/B test 2, we move the first test (A/B test 1) back to the DEV. The onboarding options (Onboarding 1 and Onboarding 2) that were used in A/B test 1 were also moved back to the QA with the possibility of editing them.
What happens when I publish 2 A/B tests to PROD?
Let's break it down in a simpler way:
Imagine we have two tests, A/B test 1 with 2 groups (Group A has Onboarding 1, Group 2 has Onboarding2) and A/B test 2 with 2 groups (Group A has Onboarding 3, Group 2 has Onboarding4). We release A/B test 1 to the PROD, and later, we release A/B test 2 to the PROD also.
After releasing A/B test 2, we move the first test (A/B test 1) back to the DEV. The onboarding options (Onboarding 1 and Onboarding 2) that were used in A/B test 1 were also moved back to the DEV without the possibility of editing them because they were already in the PROD.
What A/B tests can be finished and how?
You can finish only the tested onboardings that have been published to PROD. Thus, you should click "..." in the row of the required onboarding and opt for "Finish test." You will see a pop-up specifying the following message "Please, select Onboarding which you want to publish instead of this A/B test. The onboarding shouldn't be a variant of some other A/B test" and the drop-down menu for your obligatory onboarding choice. By clicking the relevant button you will either finish the test or proceed with its testing further.
Can I delete an A/B test?
You can only delete A/B tests on the DEV. To do this, click on the "..." in the row of the desired onboarding and choose "Delete." Once an AB test has been released to QA or Prod, it cannot be deleted.
How can you limit traffic to our AB test in your mobile app?
If you want to limit the amount of traffic that will see the onboarding through our service, you have several options:
1) You can create your own A/B testing system, and for one of the groups, launch onboarding from our service. This way, you can control how many of your users will see the onboarding created in our system. The downside of this approach is the increased waiting time for the user.
2) You can run A/B testing in our service, for one of the groups, for example, "A", create an empty onboarding ("A"), such onboarding will not be displayed but will pass control to your application. if onboarding name = "A" -> show my onboarding. Thus, this group will display your onboarding, and you will be able to regulate what percentage of people will see this onboarding.