A holdout group is a selection of random contacts who are withheld from receiving an email, or marketing treatment. The purpose of withholding from a treatment is to compare performance of the holdout group to those who were subjected to the treatment or email.
Interested in learning more about using holdout groups in your email marketing program? Contact your Account Manager for more information!
Navigate to Contacts > Email Contacts Holdout Groups. 💡You will need to select the correct list from list navigation dropdown menu.
Click New Holdout Group.
Select the Group Size - the size of the holdout group determines the percentage of contacts that will be part of the test. A treatment group of the same size will be randomly selected for comparison, however the number may become smaller if filtering is used in the messaging.
You can choose from between 1% to 50% using the slider, or up to two (2) decimal places by typing in the number in the text field.
Select a Start Date.
Enter the Test Duration - the duration of the holdout group starts at midnight on the Start Date. The Test Duration can be set to last up to ninety (90) days.
Enter any domain that would be excluded from the test (optional).
Click Save.
💡 All active or scheduled holdout groups must be completed or stopped to create a new holdout group.
Maintaining Holdout Groups
A holdout group is either Scheduled or Active, and you can Stop Test at any time. After the duration expires, the holdout group refreshes according to the pre-defined logic in the test. The expired holdout groups are listed on the Holdout Groups management page that lists the following for each completed holdout group:
Name
List Name
Start Date
Duration
End Date
Holdout Size
Export
Holdout Groups allows users to export a list of records included in a holdout group, whether the holdout group is active or expired. Scheduled holdout groups are not available for export. Upon export, two files are produced: one containing the records in the holdout group, and another containing those in the treated group.
Click the three dots in the Actions column. (For active holdout groups, click the Export button next to the Stop Test button.)
Click Export.
Override
There is an override option where users can elect to include the holdout group contacts when authoring a new message. There may be times where you determine a marketing message is important and should be sent to all contacts. Overriding the holdout group at the message level allows contacts to receive the message.
Navigate to Messages > Email New Email Broadcast or View Saved Email.
Scroll to the Advanced Message Settings.
Check the Holdout Group checkbox.
List Review Group
If specific addresses are in a holdout group but need to receive emails (e.g., internal users who need to receive all messages they would normally qualify for), they can be included in the List Review Group. Addresses in the List Review Group will not be excluded from a send if they are included in a holdout group.
The List Review Group can be found by navigating to Settings > Email Settings List Settings > Notification Settings.
Interested in learning more about Holdout Groups? Check out our video for more information: