Creating experiments using warehouse native metrics

This topic explains how to set up and configure an experiment in LaunchDarkly that uses metric events from your own data warehouse.

Prerequisites

Before you create an experiment using warehouse native metrics, you must configure your external warehouse for LaunchDarkly Experimentation.

Depending on your setup, you may also need to enable warehouse Data Export. If your experiment uses an assignment data source instead of a flag or AgentControl config, you do not need Data Export.

Configuring an experiment requires several steps:

1. Creating a warehouse native metric
2. Creating a flag or config, or setting up an assignment data source
3. Building the experiment
4. Starting an iteration

These steps are explained in detail below.

Create a metric

Warehouse native metrics must be custom metrics or ratio metrics.

Experiments using warehouse native metrics cannot use:

• Clicked or tapped metrics
• Page viewed metrics
• Metric groups
• LaunchDarkly hosted metrics that measure events from LaunchDarkly SDKs

If your experiment uses an assignment data source, your metric source and your assignment data source must use the same warehouse integration.

Create a flag or config

Before you begin an experiment, create a flag or config with the variations you plan to test the performance of. You do not need to toggle on the flag before you create an experiment, but you do have to toggle on the flag before you start an experiment iteration. Configs are on by default.

If you want to use an assignment data source instead, read the next section.

Set up an assignment data source

Instead of a flag or config, you can use an assignment data source to assign contexts to variations. LaunchDarkly reads assignment data from a table in your warehouse. These experiments do not require Data Export. Create the assignment data source before you build the experiment. To learn more, read Assignment data sources.

Limitations

You cannot run an experiment on a flag or config if:

• any of its rules have an active guarded rollout
• any of its rules have an active progressive rollout
• any of its rules are in a running Data Export experiment
• any of its rules are in a running experiment with warehouse native metrics
• the flag is a migration flag

You can build and run multiple experiments on the same flag or config, but each rule can have no more than one running experiment. You cannot run multiple experiments on the same rule at the same time.

To learn more, read Creating new flags and Creating flag variations.

Build the experiment

To build an experiment, follow the experiment creation instructions. You must select a warehouse native metric in the metric selection step.

If your experiment uses an assignment data source, you select an assignment method, an assignment source, and an experiment ID in place of a flag and its variations. To learn how, read Creating and managing assignment data sources.

Start an experiment iteration

After you create an experiment and toggle on the flag, you can start an experiment iteration in one or more environments. If your experiment uses an assignment data source instead of a flag, you can start an iteration as soon as the experiment design is complete. To learn how, read Starting and stopping experiment iterations.

If your experiment uses an assignment data source, LaunchDarkly checks how much data your configuration would query before it starts the iteration. If the volume is too large, LaunchDarkly can prevent the iteration from starting. To learn more, read Assignment data size limits.

Experiment iterations allow you to record experiments in individual blocks of time. To ensure accurate experiment results, when you make changes that impact an experiment, LaunchDarkly starts a new iteration of the experiment.