Creating holdouts

Overview

This topic explains how to create and add experiments to holdouts.

Create holdouts

Before you create a holdout, you must decide the following:

• How long to run the holdout for: we recommend anywhere from 1-3 months.
• What percentage of your customer base to include in the holdout: we recommend 1-5%.
• Whether to include all of your experiments in the holdout, or if the holdout will be specific to a certain product area.

To create a holdout:

1. Navigate to the Holdouts list from the left sidenav.
2. Click Create holdout. The “Holdout details” section appears.
3. Enter a Name.
4. Enter a Description that includes information about what kinds of experiments should be included in this holdout.
5. Enter a Holdout amount. We recommend holding out between 1-5% of your audience.
6. Click Next. The “Choose randomization unit and attributes” section appears.
7. Select a Randomization unit.
8. Add any Attributes to analyze your results by.
9. Click Next. The “Select metrics” section appears.
10. Select one or more metric or metric groups.
11. Click Next. The “Statistical analysis approach” step opens.
12. Select a statistical approach of Bayesian or Frequentist.

• If you selected a statistical approach of Bayesian, select a preset or Custom success threshold.
• If you selected a statistical approach of frequentist, select:
  • a preset or Custom significance level.
  • a One-sided or Two-sided direction of hypothesis test, depending on how confident you are as to whether the difference between the control and treatment variations will be positive or negative.

13. Click Finish.

Add experiments to holdouts

When you create a new experiment, you can decide whether to include it in an existing holdout. To learn how, read Creating experiments.

When you add an experiment to a holdout, the holdout appears as a prerequisite to the flag used in the experiment:

To include an experiment in a holdout, the experiment flag cannot have any other prerequisite flags.