A/B testing is a great way to gain this knowledge! At ConvertKit, you can test different subject lines for your broadcasts, and we'll automatically determine the winner based on their respective open rates.

A/B testing is optional and allows you to test two broadcasts subject lines against one another, within a subset of your recipients. We'll then automatically send the winning variant to the remaining recipients.

☝️ NOTE: Make sure that you take into consideration the four-hour test period before running an A/B test! It might not make sense for time-sensitive emails where four hours would be too long to wait for the full send.

To set up an A/B test for a broadcast, just click the A/B symbol next to the subject line:

This will allow you to input your two subject line variants for this email, which will be tested against one another to see which results in the higher open rate:

Adding two subject line variants is all you have to do to set up an A/B test! You can then send the broadcast as normal and the process outlined above will take place automatically — no further action required on your part (except checking back to see which was the winner, if you're curious!).

We only recommend A/B testing for emails going out to 1,000 recipients or more. A test run on a smaller number will not give actionable data, as the percentages and numbers will not be statistically significant.

For example: if you send your broadcast to 100 recipients, then each subject line variant would go out to 15 people (15% of recipients) for the initial test. Even if your email has a high open rate of ~50%, this still only means seven or eight subscribers will likely open each variant.
​
The result? That 'winning' subject line could be determined by the whims of a single person. One individual's decision of whether or not to open an email will not provide meaningful insights to apply to your greater list.

If you run an A/B test for a broadcast its reports page will have some extra statistics available.

Here's a breakdown of what you'll see stats-wise for an A/B tested broadcast:

Let's say this broadcast was sent to 1,000 recipients total. This means that each variant will have been sent to 150 recipients (15% of the total recipients list) for testing.

The aggregate stats section includes the stats for all 1,000 recipients, including the 300 recipients that received the test variants.

The results for variant A /B are the stats isolated for the test groups only (i.e. only 150 recipients each).

NOTE: Aggregate stats won't be available during the four-hour testing period, because the winner hasn't been sent to your larger list yet.

A/B testing is only available for the subject lines of broadcasts at this time.

Yes! You can cancel the test at any point. 👇🏽

Should you cancel your A/B test before it completes sending, it will behave like any completed broadcast. This means it will show you the stats for the broadcast, for the portion that was sent (30% of the recipient list). The recipient count will then update to show the small group of recipients.

At the four-hour mark of an A/B test, the system will automatically send out the subject line variant with the higher open rate at that point in time to the remaining recipients.

That winner will receive this badge in the broadcast's reports:

​

After the initial four-hour testing period, the original test recipients can (and likely will!) continue to open the original two test-sends. The isolated stats for each test-send will continue to be updated even after the test is 'over' and a winner has been determined. In some cases, what was the losing variant at the four-hour mark can end up overtaking the winner later on.

Once the testing period is over and the original winner, (the one which had the higher open rate at the four-hour mark), will have already been sent out to the remaining recipients. This winner will continue to display the 'Winner' badge, even if the other variant overtakes it later on.

Click rates do not factor into A/B testing — only open rates. We display the click rate per variant for your own reference; however, it will not factor into which subject line wins the test.

​