This topic introduces holdouts, which you can use to measure the effectiveness of your Experimentation program over time.
Holdout experiments are available with server-side and client-side SDKs, with the exception of Apex and Electron. If you use holdouts and are using a client-side SDK, your minimum SDK versions may differ from those listed on the Experimentation prerequisites page. To find the minimum required version for holdouts, read Supported features.
Holdouts let you exclude a percentage of your audience from all experiments over a set amount of time. This can help you understand how much of an impact your experiments have on certain metrics.
For example, you can hold back 5% of your user base from all of the experiments you run in a certain quarter. At the end of the quarter, you can compare the 95% that were included in experiments to the 5% that were excluded to find out which group spent more money overall, or which group signed up for services at a higher rate. If there are no measurable differences between the two groups you may want to reconsider the number, scope, and design of the experiments you’re running.
Holdout experiments and mutually exclusive experiments are similar, but are used for different reasons. Holdout experiments withhold a set of end users from all experiments. Mutually exclusive experiments use layers to ensure end users are not included in two or more specific experiments at the same time. To learn more, read Mutually exclusive experiments.
To use holdouts, you should understand the following concepts:
You also need an Admin or Owner base role, or a custom role with flag permissions, experiment permissions, and holdout permissions.
To learn how to create and manage holdouts, read Creating holdouts and Managing holdouts.
To learn how to build and run an example holdout, read Measuring Experimentation impact with holdouts.