Creating mutually exclusive experiments

Overview

This guide provides an example of how to create mutually exclusive experiments using experiment layers. Mutually exclusive experiments are two or more experiments configured to prevent contexts from being included in more than one experiment at a time.

Generally, it is not a problem to include a context in multiple experiments. Our Experimentation offering minimizes interaction effects using audience allocation, and you can be confident in the winning variation even when running multiple concurrent experiments with overlapping audiences.

However, there are some cases in which you may not want to include each context in more than one experiment at a time. For example, you may be concerned about collisions between experiments that are testing similar parts of your app, like two different changes to the same section of your app’s user interface (UI), or experiments running on both the back end and front end of the same functionality. In these cases, you can create mutually exclusive experiments using experiment layers.

Prerequisites

To complete this tutorial, you must have the following prerequisites:

• An active LaunchDarkly account with Experimentation enabled, and with permissions to create flags, edit experiments, and edit layers
• Familiarity with LaunchDarkly’s Experimentation feature
• A basic understanding of your business’s needs or key performance indicators (KPIs)

Concepts

Before you begin, you should understand the following concepts:

Layers

LaunchDarkly Experimentation achieves mutually exclusive experiments using “layers.” You can add two or more experiments to a layer to ensure that those experiments never share the same traffic.

For example, you can create a “Checkout” layer that you add all of your experiments related to the checkout process to. Then, if you run multiple experiments at once, customers are never exposed to more than one experiment variation at a time.

To learn more, read Experiment layers.

Reservation amounts

Before you create an experiment, you should decide how much of the layer’s traffic you want to include in the experiment. This is called the “reservation amount.” The more experiments you expect to include in the layer, the lower the reservation amount be.

For example:

• If you plan to have only two experiments in the layer, you might want to reserve 50% of traffic for each.
• If you plan to have five experiments in the layer, you might want to reserve 20% for each.

As new contexts encounter experiments within the layer, LaunchDarkly will assign contexts based on each experiment’s reservation amount.

In this example, each experiment will have a reservation amount of 25%. Although there are only three planned experiments for this layer, leaving an extra 25% available means you could add a fourth experiment later.

Get started

Here is an example of multiple experiments split between three different layers. The “Promotions” layer includes three experiments, the “Product pages” layer includes two experiments, and the “Checkout” layer includes three experiments:

In this example, customer A can’t be included in more than one experiment in the “Promotions” layer, more than one experiment in the “Product pages” layer, nor more than one experiment in the “Checkout” layer.

Customer A can simultaneously be included in the “Promo modal,” “Personalization at quick view,” and “Checkout flow” experiments, because those experiments are all in different layers.

This guide will walk you through creating a “Promotions” layer and the three mutually exclusive experiments within it.

Create experiment flags

To begin, create the flags you will use in your experiments within the layer.

In this example, you will create three boolean experiment flags, one for each experiment in the layer:

• the “Promo modal” flag,
• the “Promo banner” flag, and
• the “Promo at cart” flag.

To learn how, read Experiment flags.

Create metrics

Next, create or select existing metrics to use in your experiments. In this example, you will be using a clicked or tapped conversion metric to measure completed checkouts.

To create the new metric:

1. Navigate to the Metrics list.
2. Click Create metric. The “Create metric” dialog appears.
3. Select a metric kind of Clicked or tapped.
4. Enter one or more CSS selectors in the Click targets field. In this example, you are using .button.checkout.
5. Specify the Target type you want to track behavior on. In this example, you are using “Simple match.”
6. Enter the Target URL. In this example, you are using https://www.example.com/checkout.
7. Select Occurrence.
8. In the “Metric definition” section, select percentage of User units
9. Enter “Completed checkouts” as the Name for the metric.
10. Enter “Customers who complete a checkout by clicking the ‘purchase’ button” for the Description.
11. Click Create metric.

Here is what the metric looks like:

To learn more, read Clicked or tapped conversion metrics.

Create the first experiment and layer

Now, you will create the first experiment and the layer together.

Here’s how:

1. Click Create and choose Experiment. The “Create experiment” page appears.
2. Enter “Promo modal” for the Name.
3. Enter a Hypothesis.
4. Select the Feature change experiment type.
5. Click Next. The “Choose randomization unit and attributes” step opens.
6. Choose the “user” context kind from the Randomization unit menu.
7. Click Next. The “Select metrics” step opens.
8. Add the “Completed checkouts” Metric you created in the Create metrics step.
9. Click Next. The “Choose flag or AI Config variations” step opens.
10. Choose the “Promo modal” Flag you created in the Create experiment flags step.
11. Click Next. The “Set audience” step opens.
12. Check the Add experiment to layer checkbox.
13. Click Create layer.
14. Add “Promotions” as the layer Name.
15. Click Next.
16. Enter a Reservation amount of 25%.
17. In the “Define audience” section, choose which targeting rule to run the experiment on. In this example, you will use the flag’s default rule.

18. Enter the percentage of traffic you want to include in the experiment. In this example, you will include 5% of your audience.
19. Select which control variation you want LaunchDarkly to serve to the remaining population. In this example, you will select “False.”
20. Select = Split equally to divide the experiment audience between the true and false variations.
21. Click Finish. You are returned to the experiment’s Design tab.

Create the second and third experiments

To create the second and third experiments, you will repeat the process above, except you will use different flags, and you will add the experiments to the new layer you just created.

To create the second experiment:

1. Click Create and choose Experiment. The “Create experiment” page appears.
2. Enter “Promo banner” for the Name.
3. Enter a Hypothesis.
4. Select the Feature change experiment type.
5. Click Next. The “Choose randomization unit and attributes” step opens.
6. Choose the “user” context kind from the Randomization unit menu.
7. Click Next. The “Select metrics” step opens.
8. Add the “Completed checkouts” Metric.
9. Click Next. The “Choose flag or AI Config variations” step opens.
10. Choose the “Promo banner” Flag.
11. Click Next. The “Set audience” step opens.
12. Check the Add experiment to layer checkbox.
13. Click Select layer.
14. Select the “Promotions” layer from the menu.
15. Enter a Reservation amount of 25%.
16. In the “Define audience” section, choose which targeting rule to run the experiment on. In this example, you will use the flag’s default rule.
17. Enter the percentage of traffic you want to include in the experiment. In this example, you will include 5% of your audience. You can increase this percentage later, up to 100%, if needed.
18. Select which control variation you want LaunchDarkly to serve to the remaining population. In this example, you will select “False.”
19. Select = Split equally to divide the experiment audience between the true and false variations.
20. Click Finish. You are returned to the experiment’s Design tab.

To create the third experiment:

1. Click Create and choose Experiment. The “Create experiment” page appears.
2. Enter “Promo at cart” for the Name.
3. Enter a Hypothesis.
4. Select the Feature change experiment type.
5. Click Next. The “Choose randomization unit and attributes” step opens.
6. Choose the “user” context kind from the Randomization unit menu.
7. Click Next. The “Select metrics” step opens.
8. Add the “Completed checkouts” Metric.
9. Click Next. The “Choose flag or AI Config variations” step opens.
10. Choose the “Promo at cart” Flag.
11. Click Next. The “Set audience” step opens.
12. Check the Add experiment to layer checkbox.
13. Click Select layer.
14. Select the “Promotions” layer from the menu.
15. Enter a Reservation amount of 25%.
16. In the “Define audience” section, choose which targeting rule to run the experiment on. In this example, you will use the flag’s default rule.
17. Enter the percentage of traffic you want to include in the experiment. In this example, you will include 5% of your audience.
18. Select which control variation you want LaunchDarkly to serve to the remaining population. In this example, you will select “False.”
19. Select = Split equally to divide the experiment audience between the true and false variations.
20. Click Finish. You are returned to the experiment’s Design tab.

Toggle on the flags

When you are ready to run your experiments, toggle On the related flags.

Start experiment iterations

The last step is to start an iteration for each of your mutually exclusive experiments.

To start your experiment iterations:

1. Navigate to the Experiments list.
2. Click on “Promo modal.”
3. Click Start.
4. Repeat steps 1-3 for the remaining two experiments.

You are now running three mutually exclusive experiments, with no overlap between the contexts included in each.

Read results

When you are finished running your experiments, you can look at which variation has the highest probability to beat control and probability to be best, or the highlighted p-value, to decide which features to ship to your customers. To learn more, read Winning variations.

Conclusion

In this guide, you learned how to create layers to prevent contexts from being included in multiple concurrent experiments.

Mutually exclusive experiments are not limited to two or three at a time. When it’s time to create your own, you can create as many mutually exclusive experiments as you need to prevent collisions between experiments running on similar parts of your app or infrastructure.

To get started building your own experiment, follow our Quickstart for Experimentation.