Mutually exclusive experiments

Overview

This topic explains how to use layers to set up mutually exclusive experiments in LaunchDarkly. Mutually exclusive experiments are two or more experiments configured to prevent contexts from being included in more than one of them at a time.

Usually, it is not a problem to include a context in multiple experiments. However, there are some cases in which you may not want to include a context in more than one experiment at a time. For example, you may be concerned about collisions between experiments that are testing similar parts of your app, like two different changes to the same section of your app’s user interface (UI), or experiments running on both the back end and front end of the same functionality. In this case you can eliminate the interaction effect between experiments using layers.

Experiment layers

LaunchDarkly Experimentation achieves mutually exclusive experiments using “layers.” You can add two or more experiments to a layer to ensure that those experiments never share the same traffic.

For example, you can create a “Checkout” layer that you add all of your experiments related to the checkout process to. Then, if you run multiple experiments at once, customers are never exposed to more than one experimental variation at a time.

Here is an example of three different layers: a “Promotions” layer with three experiments, a “Product pages” layer with two experiments, and a “Checkout” layer with three experiments:

In the example, customer A can’t be included in more than one experiment in the “Promotions” layer, more than one experiment in the “Product pages” layer, nor more than one experiment in the “Checkout” layer. Customer A can simultaneously be included in the “Promo modal,” “Personalization at quick view,” and “Checkout flow” experiments, because those experiments are all in different layers.

Reservation amounts

Before you create an experiment, decide how much of the layer’s traffic you want to include in the experiment. This is called the “reservation amount.” The more experiments you expect to include in the layer, the lower the reservation amount be.

For example:

• If you plan to have only two experiments in the layer, you might want to reserve 50% of traffic for each.
• If you plan to have five experiments in the layer, you might want to reserve 20% for each.

As new contexts encounter experiments within the layer, LaunchDarkly will assign contexts based on each experiment’s reservation amount. You can change an experiment’s reservation amount on the experiment’s Design tab.

In the below example:

• 25% of a layer’s contexts are reserved for a specific experiment
• of that 25%, 21% of the audience is included in the experiment, and 4% is excluded from the experiment and served the control variation
• of the 21% in the experiment, 7% is assigned to each variation.

Here is an illustration of the audience allocation:

Create new layers

When you create an experiment, you can either:

• add the experiment to an existing layer, or
• create a new layer, and add the experiment to that new layer.

To create a new layer:

1. Follow the directions to create a new experiment.
2. In the “Set audience” step, check the Add experiment to layer checkbox.
3. Click Create layer.
4. Add a Name.
5. Add a Description.

6. Click Create layer.
7. Add a Reservation amount.
8. Continue to the “Define audience” step and finish creating the experiment.

This new layer is now available to add additional experiments to.

You can also use the REST API: Create layer

Add experiments to existing layers

To add a new experiment to an existing layer:

1. Follow the directions to create a new experiment.
2. In the “Set audience” step, check the Add experiment to layer checkbox.
3. Click Select layer.
4. Select an existing layer from the layer menu. You can view all of the existing layers and the experiments within them at this step.

5. Add a Reservation amount.
6. Continue to the “Define audience” step and finish creating the experiment.

You can also use the REST API: Create experiment

Remove experiments from layers

To remove an experiment from a layer:

1. Navigate to the Design tab of your experiment.
2. Click Edit experiment.
3. Navigate to the “Audience” section and click the pencil icon.
4. Uncheck the Add experiment to layer checkbox.
5. Click Save.

Before you can archive an experiment, you must remove it from any layers.