Alia’s A/B testing analytics dashboard helps you confidently identify winning popup variations by measuring signup performance, conversion impact, and downstream revenue.
With advanced filtering capabilities, you can analyze results across specific shopper segments — including UTMs, device type, country, survey responses, browsing behavior, and more — allowing you to uncover insights that go far beyond surface-level metrics.
Step 1: Access A/B Test Results
Navigate to your campaign dashboard in Alia.
Click on "View A/B Test Results" to open the test results page.
You’ll land on the analytics dashboard where all experiment data is automatically aggregated.
Reviewing a past test? To see the actual popup versions used in a completed test, open the past test and use the dropdown near the "View test results" section. This lets you compare what changed between variants after the fact.
Step 2: Review Test Timeline
The dashboard displays the full test duration, including:
Start date
End date (or current progress if still running)
If your test is active, metrics update in near real-time so you can monitor performance trends as data accumulates.
💡 Best practice: Avoid ending tests too early — stronger data leads to more reliable winners.
Step 3: Compare Control vs. Test Group
Each experiment includes:
Control: Your original popup
Variants: The versions being tested
You can have an unlimited number of variants
Review performance side-by-side to understand which experience drives stronger results.
👉 The percentage lift displayed between variants helps quickly identify directional winners.
Connecting variants to different lists: each A/B test variant can be connected to a different email or SMS list. Test and validate the setup before drawing conclusions from the results.
Step 4: Segment Your Results with Advanced Filters
One of the most powerful features of Alia’s new analytics experience is the ability to filter performance across virtually any shopper dimension.
You can analyze results by:
Traffic & Marketing
UTM source, medium, and campaign
Initial path
Domain
Shopper Attributes
Country, region, and city
Shopify market and locale
Device type, browser, and OS
Behavior & Actions
Visitors matching targeting
Email submissions
Reward variants
Purchased products or collections
Zero-Party Data
Survey responses
Mini-quiz answers
Custom user properties
💡 Why this matters: A test may appear neutral at a high level but reveal a clear winner within a high-value segment — such as paid traffic or mobile shoppers.
This is where the biggest optimization opportunities often live.
💡 The percentage difference between the control and test groups will help determine which performs better.
Step 5: Evaluate Key Performance Metrics
Your dashboard includes all primary analytics, plus sitewide impact metrics.
Core Metrics to Watch
Email & SMS submit rates
Opt-in rates
Attributed conversion rate
Attributed orders and sales
Average attributed order value
Time to purchase
Sitewide Impact Metrics
Sitewide conversion rate
Sitewide sales
Bounce rate
These help ensure your popup improves signup performance without harming the overall shopping experience.
Attributed and Sitewide metrics use different anchor points. Attributed metrics are measured from the moment of signup, while Sitewide metrics are measured from the visitor's first visit to your site. Because of this, the two sets of numbers will not line up even when you select the same window length. See Understanding Your Alia Statistics for a full explanation.
💡 If statistical significance isn’t reached, allow the test to run longer for more reliable results.
Step 6: Check Statistical Significance
Each metric includes a significance indicator:
✅ Check mark — Results are statistically significant
⏳ Clock icon — More data is needed
💡 Pro Tip: If significance hasn’t been reached, allow the test to run longer before making a decision. Ending tests prematurely can lead to false winners.
Why early results can be misleading
Early in a test, results are noisy and a weaker variant can appear to be winning. This is normal, not a sign something is wrong.
Metrics also stabilize at different speeds. Bounce rate draws on a much larger volume of data than conversion rate, so it settles sooner. If bounce rate and conversion rate appear to disagree early on, that is usually a data-volume difference rather than a real conflict. Let the test run longer before deciding.
Step 7: Identify and Publish the Winning Variation
Once a clear winner emerges:
Select the top-performing variant
Publish it directly from the results page
Alia will automatically:
✅ Keep the winning popup live
✅ Move the losing variation to draft
✅ Preserve test data for future analysis
What’s Next?
Winning a test is just the beginning.
Use your insights to continuously improve performance:
🚀 Test new messaging or offers
🎨 Experiment with design changes
🎯 Refine targeting
📊 Optimize by traffic source
🧠 Personalize experiences using zero-party data
The highest-performing brands treat A/B testing as an ongoing growth engine — not a one-time exercise.