Our service allows you to create A/B tests for testing hypotheses of your iOS application without releasing it in the AppStore. In this article, you will find instructions on how to create and configure A/B tests.
Creating an A/B Test
To create an A/B test, select the "A/B tests" tab on the control panel.
Here you will find the "+ ADD TEST" option to create a test and a panel displaying information about the tests: "Name," "Current environment," "Variants," and "Actions."
Important: Before creating an A/B test, make sure you are in the correct project.
Click on +ADD TEST to create a test. A window will open for setting up the A/B test.
Here you can:
Name the test
Add or remove variants
Select onboardings for different variants
Adjust the distribution of users to each variant using sliders
The "Done" button closes the screen and saves the test in the DEV environment.
The "Release to QA" button publishes the test in the QA environment for internal testing and allows you to publish the test in Production for AppStore users.
The "Cancel" button closes the screen without saving the A/B test.
In the "A/B tests" dashboard, the "Current env" column conveniently displays the environment of each A/B test, aiding in easy tracking and management across different environments.
Releasing A/B Tests
Publishing to QA for Internal Testing
After setting up your A/B test, you can publish it to the QA environment for internal testing. You can do this in 3 ways:
Immediately after creating an A/B test, you can click on "Release to QA" in the pop-up that appears. Use this method when you are ready to move your test from DEV to QA immediately after setting it up.
2. Click on "..." in the "Actions" column of your A/B test. Select "Settings" from the dropdown menu. In the settings pop-up, click on "Release to QA".
3. Click on "..." in the "Actions" column of your A/B test and click Publish to QA.
When you choose 'Release to QA', a confirmation pop-up appears. If you decide not to proceed, simply click "No". This action will cancel the publishing process. To publish the test to QA, click "Yes"
If you have published onboarding in QA, you cannot change its name, add variants, or change onboardings in variants. But you can change the percentage distribution.
To change the distribution:
Click on "..." in the "Actions" column of your A/B test
Click "Settings".
Adjust the distribution using sliders for each variant
Click "Done" to save the changes.
Click "Done" to save your changes.
Publishing to Prod for AppStore Users
When publishing your A/B test to the Production (Prod) environment, it becomes active and available to your AppStore users. You can do this in 3 ways:
Immediately after creating an A/B test, you can click on "Release to QA", then on "Release to Prod". This method is useful if you prefer to skip the QA phase and release to Prod immediately after creating an A/B test.
2. Click on "..." in the "Actions" column of your A/B test. Select "Settings" from the dropdown menu. In the settings pop-up, click on "Release to Prod".
3. Click on "..." in the "Actions" column of your A/B test and click Publish to Prod.
When you choose 'Release to Prod', a confirmation pop-up appears. If you decide not to proceed, simply click "No". This action will cancel the publishing process. To publish the test to Prod, click "Yes".
If you have published onboarding in Prod, you cannot change its name, add variants, or change onboardings in variants. But you can change the percentage distribution.
To change the distribution:
Click on "..." in the "Actions" column of your A/B test
Click "Settings".
Adjust the distribution using sliders for each variant
Click "Done" to save the change.
In the DEV environment, there's no limit to the number of tests you can add, but only one A/B test at a time can be released in the QA or Prod environments.
Finishing the A/B test:
If you have published onboarding in Prod, conducted the test, and decided to conclude it, click "..." next to the onboarding and select "Finish test".
A pop-up will appear, prompting you to "Please, select Onboarding which you want to publish instead of this A/B test."
In the dropdown list, you can choose:
An onboarding that participates in the current A/B test
Any created in the current project.
Deleting A/B Tests
A/B tests can only be deleted in the DEV environment. To do this, click "..." next to the desired onboarding and choose "Delete".
Note that deleting an A/B test does not remove the actual onboarding.
The article with Questions and Answers about AB tests you can find here