Creating mutually exclusive experiments

Overview

This guide provides an example of how to create mutually exclusive experiments using experiment layers. Mutually exclusive experiments are two or more experiments configured to prevent contexts from being included in more than one experiment at a time.

Generally, it is not a problem to include a context in multiple experiments. Our Experimentation offering minimizes interaction effects using audience allocation, and you can be confident in the winning variation even when running multiple concurrent experiments with overlapping audiences.

However, there are some cases in which you may not want to include each context in more than one experiment at a time. For example, you may be concerned about collisions between experiments that are testing similar parts of your app, like two different changes to the same section of your app’s user interface (UI), or experiments running on both the back end and front end of the same functionality. In these cases, you can create mutually exclusive experiments using experiment layers.

Prerequisites

To complete this tutorial, you must have the following prerequisites:

• An active LaunchDarkly account with Experimentation enabled, and with permissions to create flags, edit experiments, and edit layers
• Familiarity with LaunchDarkly’s Experimentation feature
• A basic understanding of your business’s needs or key performance indicators (KPIs)

Concepts

Before you begin, you should understand the following concepts:

Layers

LaunchDarkly Experimentation achieves mutually exclusive experiments using “layers.” You can add two or more experiments to a layer to ensure that those experiments never share the same traffic.

For example, you can create a “Checkout” layer that you add all of your experiments related to the checkout process to. Then, if you run multiple experiments at once, customers are never exposed to more than one experiment variation at a time.

To learn more, read Experiment layers.

Reservation amounts

Before you create an experiment, you should decide how much of the layer’s traffic you want to include in the experiment. This is called the “reservation amount.” The more experiments you expect to include in the layer, the lower the reservation amount be.

For example:

• If you plan to have only two experiments in the layer, you might want to reserve 50% of traffic for each.
• If you plan to have five experiments in the layer, you might want to reserve 20% for each.

As new contexts encounter experiments within the layer, LaunchDarkly will assign contexts based on each experiment’s reservation amount.

In this example, each experiment will have a reservation amount of 25%. Although there are only three planned experiments for this layer, leaving an extra 25% available means you could add a fourth experiment later.

Get started

Here is an example of multiple experiments split between three different layers. The “Promotions” layer includes three experiments, the “Product pages” layer includes two experiments, and the “Checkout” layer includes three experiments:

In this example, customer A can’t be included in more than one experiment in the “Promotions” layer, more than one experiment in the “Product pages” layer, nor more than one experiment in the “Checkout” layer.

Customer A can simultaneously be included in the “Promo modal,” “Personalization at quick view,” and “Checkout flow” experiments, because those experiments are all in different layers.

This guide will walk you through creating a “Promotions” layer and the three mutually exclusive experiments within it.

Create experiment flags

To begin, create the flags you will use in your experiments within the layer.

In this example, you will create three boolean experiment flags, one for each experiment in the layer:

• the “Promo modal” flag,
• the “Promo banner” flag, and
• the “Promo at cart” flag.

To learn how, read Experiment flags.

Create metrics

Next, create or select existing metrics to use in your experiments. In this example, you will be using a clicked or tapped conversion metric to measure completed checkouts.

To create the new metric:

1. Navigate to the Metrics list.
2. Click Create metric. The “Create metric” dialog appears.
3. Select a metric kind of Clicked or tapped.
4. Enter one or more CSS selectors in the Click targets field. In this example, you are using .button.checkout.
5. Specify the Target type you want to track behavior on. In this example, you are using “Simple match.”
6. Enter the Target URL. In this example, you are using https://www.example.com/checkout.
7. Select Occurrence.
8. In the “Metric definition” section, select percentage of User units
9. Enter “Completed checkouts” as the Name for the metric.
10. Enter “Customers who complete a checkout by clicking the ‘purchase’ button” for the Description.
11. Click Create metric.

Here is what the metric looks like:

To learn more, read Clicked or tapped conversion metrics.

Create the first experiment and layer

Now, you will create the first experiment and the layer together.

Here’s how:

1. Click Create and choose Experiment. The experiment Design tab appears.
2. Select theFeature change experiment type.
3. Enter “Promo modal” for the Name.
4. Enter a Hypothesis.
5. Add the “Completed checkouts” Metric you created in the Create metrics step.
6. Choose the “Promo modal” Flag you created in the Create experiment flags step.
7. Choose “user” as the randomization unit to Target by.
8. Choose the default targeting rule for the Experiment audience.

9. Click Add experiment to exclusion layer.
10. Click Create layer.
11. Add “Promotions” as the layer Name.
12. Enter a Reservation amount of 25%.
13. Click Save layer.
14. Select the Sample size for the experiment. In this example, you will include 5% of your audience.
15. Choose “False” for the Variation served to users outside this experiment.
16. Select Split equally for the variations split.
17. Select a Statistical approach of Bayesian or frequentist.
    • If you selected a statistical approach of Bayesian, select a preset or Custom success threshold.
    • If you selected a statistical approach of frequentist, select:
      • a Significance level.
      • a one-sided or two-sided Direction of hypothesis test.

18. Scroll to the top of the page and click Save.

Create the second and third experiments

To create the second and third experiments, you will repeat the process above, except you will use different flags, and you will add the experiments to the new layer you just created.

To create the second experiment:

1. Click Create and choose Experiment. The experiment Design tab appears.
2. Select theFeature change experiment type.
3. Enter “Promo banner” for the Name.
4. Enter a Hypothesis.
5. Add the “Completed checkouts” Metric.
6. Choose the “Promo banner” Flag.
7. Choose “user” as the randomization unit to Target by.
8. Choose the default targeting rule for the Experiment audience.
9. Click Add experiment to exclusion layer.
10. Click Select layer.
11. Select the “Promotions” layer from the menu.
12. Enter a Reservation amount of 25%.
13. Click Save layer.
14. Select Sample size of 5%.
15. Choose “False” for the Variation served to users outside this experiment.
16. Select Split equally for the variations split.
17. Select the same Statistical approach you chose for the first and second experiment.
18. Scroll to the top of the page and click Save.

To create the third experiment:

1. Click Create and choose Experiment. The experiment Design tab appears.
2. Select theFeature change experiment type.
3. Enter “Promo at cart” for the Name.
4. Enter a Hypothesis.
5. Add the “Completed checkouts” Metric.
6. Choose the “Promo at cart” Flag.
7. Choose “user” as the randomization unit to Target by.
8. Choose the default targeting rule for the Experiment audience.
9. Click Add experiment to exclusion layer.
10. Click Select layer.
11. Select the “Promotions” layer from the menu.
12. Enter a Reservation amount of 25%.
13. Click Save layer.
14. Select Sample size of 5%.
15. Choose “False” for the Variation served to users outside this experiment.
16. Select Split equally for the variations split.
17. Select the same Statistical approach you chose for the first experiment.
18. Scroll to the top of the page and click Save.

Toggle on the flags

When you are ready to run your experiments, toggle On the related flags.

Start experiment iterations

The last step is to start an iteration for each of your mutually exclusive experiments.

To start your experiment iterations:

1. Navigate to the Experiments list.
2. Click on “Promo modal.”
3. Click Start.
4. Repeat steps 1-3 for the remaining two experiments.

You are now running three mutually exclusive experiments, with no overlap between the contexts included in each.

Read results

When you are finished running your experiments, you can look at which variation has the highest probability to be best, or the highlighted p-value, to decide which features to ship to your customers. To learn more, read Analyzing experiments.

Conclusion

In this guide, you learned how to create layers to prevent contexts from being included in multiple concurrent experiments.

Mutually exclusive experiments are not limited to two or three at a time. When it’s time to create your own, you can create as many mutually exclusive experiments as you need to prevent collisions between experiments running on similar parts of your app or infrastructure.

To get started building your own experiment, follow our Quickstart for Experimentation.