A/B Testing with Rebuy allows you to test different versions of offers and product recommendations across multiple customer touchpoints: the Checkout page, Thank You page, and Order Status page. By testing variations, you can optimize post-purchase engagement, maximize revenue, and increase conversions. The process is simple, taking under 5 minutes to set up, and once the experiment concludes, the winning variation is automatically deployed for all incoming traffic.
This guide walks you through the steps to set up A/B tests for Checkout Offers, Thank You Page Offers, and Order Status Page Offers, so you can optimize your store's performance across different stages of the customer journey.
Checkout Offers Compatibility Matrix
Feature Description | Shopify Plus | Non-Shopify Plus |
Product Recommendations - Checkout | Yes | No |
Content Blocks - Checkout | Yes | No |
Content Blocks - Thank You Page | Yes | Yes |
Content Blocks - Order Status | Yes | Yes |
Product Recommendations - Post-Purchase | Yes | Yes |
Product Recommendations - Thank You Page | Yes | Yes |
Product Recommendations - Order Status | Yes | Yes |
Step-by-Step Guide to Building a Checkout Offer A/B Test
Step 1: Create the Offer Widget or Content Block
Before setting up your A/B test, create at least two Rebuy widgets or content blocks (or one widget if you’re testing against a "no widget" scenario). These will be the elements tested in the experiment.
Product Recommendations: You can add product recommendations (e.g., related products, upsells, or cross-sells) as part of your offer.
Checkout Widgets: Available to Shopify Plus merchants only.
Thank You Page Widgets: Available to all Shopify plans
Order Status Page Widgets: Available to all Shopify plans
Content Blocks: Available to all merchants and can be used on the Checkout, Thank You, and Order Status pages. However, Shopify Plus plans are the only plan that can place them directly on the checkout page.
If you haven’t created any checkout widgets yet, refer to our documentation for instructions on building them. These are available for Shopify Plus only.
If you haven’t created any Thank You Page or Order Status Page widgets yet, refer to our documentation for instructions on building them.
If you haven’t created any content blocks yet, refer to our documentation for instructions on building them.
Please note that Checkout widgets are exclusively available to Shopify Plus merchants due to Shopify's limitations. As a result, only Shopify Plus users can place widgets directly in the checkout process. However, Non-Shopify Plus merchants can still use content blocks, thank you page, and order status page widgets.
Step 2: Access A/B Testing
Open the Rebuy app and go to the main menu.
Select A/B Testing.
Click the Create New Experiment button.
In the pop-up, enter a descriptive name for your experiment (for easy tracking).
Choose the Experiment Type:
Checkout Offers: Use this A/B test type for checkout, thank you page, or order status page widgets.
Click Add Experiment to enter the experiment editor.
Step 3: Set Control and Variations
On the experiment settings page, select your control widget (the baseline for comparison).
Add up to 9 widget variations that you want to test. These could include different product recommendations, discount offers, or content blocks.
Set the display percentage for each widget (control and variations). This determines how often each variation will be shown to users during the experiment.
Step 4: Select the Experiment Goal
Scroll down to the "Experiment Goal" section.
Choose either "Revenue" (default) or "Conversion Rate" as the primary metric for determining the winner:
Revenue: The variation that generates the most revenue wins. This goal is ideal if you’re optimizing for total income.
Conversion Rate: The variation with the highest percentage of order conversions wins. This goal is recommended if you’re focusing on improving the number of completed purchases.
Step 5: Set the Start and End Time (Optional)
You can schedule the experiment to start and end at specific times or run it manually:
In the Test Duration section, set your Start Date and Start Time.
Set the End Date and End Time to automatically end the experiment at a specified time.
If you prefer to manually start and stop the experiment, skip this step.
Note: A/B tests cannot be scheduled to start more than 30 days in advance or end more than 30 days after the start date.
Step 6: Save the Experiment
Once all required elements have been configured, the “Save Experiment” button will become active. Click it to finalize your experiment setup.
Step 7: Get the Placeholder ID
After saving, your experiment will be in Draft mode.
Find the Placeholder ID on the right side of the editor.
Use this Placeholder ID to install the widget on the relevant page (Checkout, Thank You, or Order Status). This ensures that the A/B test runs correctly.
Important: Do not install the original widget IDs directly—only use the Placeholder ID to prevent errors.
Important:
Do not directly install the original widget IDs for your control and variations. Doing so will cause them to be displayed simultaneously, invalidating your A/B test.
The Placeholder ID is essential for the test to function correctly by dynamically showing the appropriate widget variation.
Step 8: Install the Placeholder ID
Installing the Placeholder ID via Shopify's checkout editor
Open the checkout editor in your Shopify portal.
Add Rebuy Engine app block.
Add the Placeholder ID directly into the app block.
For more detailed instructions on installing via app blocks, refer to Rebuy’s app blocks help documentation.
Please note: If you install the placeholder ID in your checkout and your A/B test is not yet live, an error message will appear stating: Invalid Widget ID (incorrect type or possibly deleted): Verify your widgets before updating extension in the Checkout Editor
Once you start your A/B test and refresh the checkout editor page, the error message will disappear, and a new one will appear stating: Experiment: The selected variant will display here
Starting an Experiment
Option 1: Start from the Edit Screen
If you remain on the experiment’s edit screen after saving it, you can start the experiment immediately.
Locate the "Start Experiment" button at the top right-hand side of the experiment editor (in the main heading).
Click "Start Experiment." A pop-up confirmation will appear.
Click "Start Experiment" again to confirm, and your A/B test will go live.
Option 2: Start from the Dashboard
If you leave the edit screen, you can start the experiment from the Drafts tab on the A/B Testing dashboard.
Find the experiment you want to start and click the vertical ellipsis (three dots) next to the "Draft" label.
Select "Start Experiment" from the menu. A pop-up confirmation will appear.
Click "Start Experiment" again to confirm, and your A/B test will go live.
Ending Your Experiment
Scheduled End Time
If you configured a scheduled end time during setup, the experiment will automatically conclude based on that schedule.
Manually Ending the Experiment
If you wish to end the experiment early, go to the Active tab on the A/B Testing dashboard.
Click the vertical ellipsis (three dots) next to the active experiment.
Select "End Experiment" to immediately stop the test, or "Edit" to adjust the scheduled end time.
After the Experiment Ends
Once the experiment concludes, the results will be available under the Completed tab on the A/B Testing dashboard.
The winning widget configuration will automatically replace the placeholder in your store, making it permanently visible to customers without any further action required.
Monitoring Experiment Performance
Monitoring Active Experiments
After an experiment starts, you can view it under the Active tab on the A/B Testing page within Manage Your Experiments.
Use the vertical ellipsis menu to end the experiment immediately or edit it to modify the scheduled end time if desired.
Viewing Performance Metrics
Summary performance metrics for both active and completed experiments are visible within their respective tabs.
Metrics like revenue, conversion rate, and engagement help you assess how well each variation performs against the control.
Summary and Best Practices
Starting the Experiment: Choose either the edit screen or the dashboard to launch your experiment. Both methods offer a straightforward way to get your test live.
Managing and Ending the Experiment: Utilize the scheduling features or manually end the test using the dashboard tools.
Automatic Deployment of Winning Widget: Rebuy’s system ensures that once a test concludes, the winning widget is deployed automatically, optimizing your store’s performance seamlessly.
FAQs
Q: Can I A/B test content blocks with Rebuy?
A: Yes, you can A/B test content blocks, but it’s important to be aware of certain limitations and requirements based on your Shopify plan.
Q: How can I validate which Widget ID is rendering on the Checkout page?
A: To confirm which Widget ID is rendering on the Checkout Page, open your browser's Developer Tools, then click on the Console tab. You should then see a validation message which contains the Experiment ID and Widget ID. The Selected ID is the Rebuy Widget ID that is rendering on the page via the experiment:
Q: Can I A/B test content blocks on the Thank You or Order Status pages?
A: Yes, all merchants, including non-Shopify Plus merchants, can A/B test content blocks on the Thank You and Order Status pages. These pages are a great place to show personalized product recommendations, promotions, or offers to engage customers after they complete a purchase.
Q: Can I A/B test offers in Checkout?
A: Yes, Shopify Plus merchants can A/B test offers directly in the Checkout process using Checkout widgets. These widgets allow you to test different product recommendations, discounts, or personalized offers during the checkout flow.
For non-Shopify Plus merchants, Shopify's limitations prevent placing widgets directly in the checkout. However, you can still test offers on the Thank You page and Order Status page.
Q: Can I A/B test Rebuy Monetize?
A: Yes, you can A/B test Rebuy Monetize offers in your store! But it can only be A/B tested on the Thank You Page or Order Status Pages.