Creating Data Export experiments

Overview

This topic explains how to set up and configure an experiment in LaunchDarkly that you plan to analyze in an external tool.

Data Export experiments let you export experiment data to a third-party tool for analysis using Data Export. Data Export experiments do not require a LaunchDarkly metric, and do not generate analysis results in the LaunchDarkly user interface (UI). This experiment type is useful when you want to use LaunchDarkly to manage your experiment, but have a preferred external analysis tool that allows you to import experiment data from other systems.

Configuring a Data Export experiment requires the following steps:

1. Enabling Data Export
2. Creating the flag or AI Config and its variations,
3. Building the experiment
4. Turning on the flag or AI Config
5. Starting an iteration
6. You will then automatically begin receiving experiment data in your Data Export files.

These steps are explained in detail below.

Prerequisites

Before you build a Data Export experiment, you must:

• enable Data Export in your LaunchDarkly account
• understand randomization units

Create flags or AI Configs

Before you begin an experiment, create a flag or AI Config with the variations you plan to test the performance of. You do not need to toggle on the flag before you create an experiment, but you do have to toggle on the flag before you start an experiment iteration. AI Configs are on by default.

To learn more, read Creating new flags, Creating flag variations, Create AI Configs, and Create and manage AI Config variations.

You cannot run a Data Export experiment on a flag if:

• the flag has an active guarded rollout
• the flag has an active progressive rollout
• the flag is already in a running experiment
• the flag is a migration flag

You can build multiple Data Export experiments on the same flag or AI Config, but you can run only one of those experiments at a time.

Build experiments

You can view all of the experiments in your environment on the Experiments list.

To build an experiment:

1. Click Create and choose Experiment. The “Create experiment” dialog appears.
2. Enter an experiment Name.
3. Enter a Hypothesis.
4. Click Create experiment. The experiment Design tab appears.
5. Select the Data Export only experiment type.
6. Choose a context kind to Randomize by.
7. Choose a Flag or AI Config to use in the experiment.
   • Click Create flag or Create AI Config to create and use a new flag or AI Config.
8. Choose a targeting rule for the Experiment audience.
   • If you want to restrict your experiment audience to only contexts with certain attributes, create a targeting rule on the flag or AI Config you include in the experiment and run the experiment on that rule.
   • If you don’t want to restrict the audience for your experiment, run the experiment on the default rule. If the flag or AI Config doesn’t have any targeting rules, the default rule will be the only option.

9. (Optional) If you want to exclude contexts in this experiment from certain other experiments, click Add experiment to exclusion layer and select a layer.

10. Choose the Variation served to users outside this experiment. Contexts that match the selected targeting rule but are not in the experiment will receive this variation.
11. Select the Sample size for the experiment. This is the percentage of all of the contexts that match the experiment’s targeting rule that you want to include in the experiment.
12. (Optional) Click Advanced to edit variation reassignment. For most experiments, we recommend leaving this option on its default setting. To learn more, read Carryover bias and variation reassignment.
13. (Optional) Click Edit to update the variation split for contexts that are in the experiment.
    • You can Split equally between variations, or assign a higher percentage of contexts to some variations than others.
    • Click Save audience split.
14. Select a variation to serve as the Control.
15. (Optional) If you want to include the experiment in a holdout, click Advanced, then select a Holdout name.

16. Scroll to the top of the page and click Save.

If needed, you can save your in-progress experiment design to finish later. To save your design, click Save at the top of the creation screen. Your in-progress experiment design is saved and appears on the Experiments list. To finish building the experiment, click on the experiment’s name and continue editing.

After you have created your experiment, the next step is to toggle on the flag. AI Configs are on by default. Then, you can start an iteration.

Turn on flags or AI Configs

For an experiment to begin recording data, the flag or AI Config used in the experiment must be on. Targeting rules for AI Configs are on by default. To learn how to turn targeting rules on for flags, read Turning flags on and off.

Start experiment iterations

After you create an experiment and the flag or AI Config is toggled on, you can start an experiment iteration in one or more environments.

To start an experiment iteration:

1. Navigate to the Experiments list.
2. Click on the environment section containing the experiment you want to start.
   • If the environment you need isn’t visible, click the + next to the list of environment sections. Search for the environment you want, and select it from the list.

3. Click on the name of the experiment you want to start an iteration for. The Design tab appears.
4. Click Start.
5. Repeat steps 2-4 for each environment you want to start an iteration in.

Experiment iterations allow you to record experiments in individual blocks of time. To ensure accurate experiment results, when you make changes that impact an experiment, LaunchDarkly starts a new iteration of the experiment.

To learn more about starting and stopping iterations, read Starting and stopping experiment iterations..

Each time you start an iteration, you will automatically begin receiving experiment data in your Data Export files.