For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Sign inTry it free
DocsGuidesSDKsIntegrationsAPI docsTutorialsFlagship blog
DocsGuidesSDKsIntegrationsAPI docsTutorialsFlagship blog
  • Get started
    • Overview
    • Onboarding
    • Get started
    • Launch Insights
    • LaunchDarkly architecture
    • LaunchDarkly vocabulary
  • AgentControl
    • AgentControl
    • Manage AgentControl
  • Feature flags
    • Create flags
    • Target with flags
    • Flag templates
    • Manage flags
    • Code references
    • Contexts
    • Segments
  • Releases
    • Releasing features with LaunchDarkly
    • Release policies
    • Percentage rollouts
    • Progressive rollouts
    • Guarded rollouts
    • Feature monitoring
    • Release pipelines
    • Engineering insights
    • Release management tools
    • Applications and app versions
    • Change history
    • Restoring previous flag versions
  • Observability
    • Observability
    • Session replay
    • Error monitoring
    • Logs
    • Traces
    • Observability metrics
    • Product analytics events
    • LLM observability
    • Alerts
    • Dashboards
    • Service map
    • Vega for auto-remediation
    • Observability MCP server
    • Search specification
    • Observability settings
    • Observability integrations
  • Experimentation
    • Experimentation
    • Experiment metric types
    • Experiment configuration
    • Managing experiments
    • Analyzing experiments
    • Multi-armed bandits
      • Creating multi-armed-bandits
      • Editing and archiving multi-armed bandits
      • Multi-armed bandit results
    • Holdouts
  • Metrics and events
    • Metrics in LaunchDarkly
    • Creating metrics
    • Metric groups
    • Events
    • Autogenerated metrics
  • Warehouse native
    • Warehouse native metrics
    • Setting up external warehouses
    • Creating experiments using warehouse native metrics
  • Infrastructure
    • Connect apps and services to LaunchDarkly
    • LaunchDarkly in China and Pakistan
    • LaunchDarkly in the European Union (EU)
    • LaunchDarkly in federal environments
    • Public IP list
  • Your account
    • Projects
    • Views
    • Environments
    • Tags
    • Teams
    • Members
    • Roles
    • Account security
    • Feature previews
    • Billing and usage
    • Changelog
Sign inTry it free
LogoLogo
On this page
  • Prerequisites
  • Create flags or AgentControl configs
  • Create metrics
  • Build multi-armed bandits
  • Turn on flags or AgentControl configs
  • Start iterations
ExperimentationMulti-armed bandits

Creating multi-armed bandits

Was this page helpful?
Previous

Editing and archiving multi-armed bandits

Next
Built with

This topic explains how to set up and configure a multi-armed bandit (MAB) in LaunchDarkly.

Configuring a MAB requires several steps:

  1. Creating the flag or AgentControl config and its variations,
  2. Creating a metric,
  3. Building the multi-armed bandit,
  4. Turning on the flag or AgentControl config, and
  5. Starting an iteration.

These steps are explained in detail below.

Prerequisites

Before you build a MAB, you should read about and understand the following concepts:

  • Multi-armed bandits
  • randomization units

Create flags or AgentControl configs

Before you begin a MAB, create a flag or AgentControl config with the variations you plan to optimize the performance of. You do not need to toggle on the flag before you create the MAB, but you do have to toggle flags on before you start an iteration. AgentControl configs are on by default.

To learn more, read Creating new flags, Creating flag variations, Create configs, and Create and manage config variations.

You cannot run a MAB on a flag if:

  • the flag has an active guarded rollout
  • the flag has an active progressive rollout
  • the flag is in a running Data Export experiment
  • the flag is in a running experiment with warehouse native metrics
  • the flag is a migration flag

You can build and run multiple MABs on the same flag or AgentControl config as long as there is only one running MAB per targeting rule. You cannot run multiple MABs on the same rule at the same time.

Create metrics

Metrics measure audience behaviors affected by your flags. You can use metrics to track all kinds of things, from how often end users access a URL to how long that URL takes to load a page. You can reuse existing metrics in multiple MABs, or create new ones for your MABs.

To learn how to create your own new metric, read Metrics. LaunchDarkly also automatically creates metrics for AgentControl configs. To learn more, read AgentControl config autogenerated metrics.

Multi-armed bandits do not support metrics using percentile analysis methods

Multi-armed bandits support metrics using the “Average” metric analysis method. You cannot use metrics using a percentile analysis method with a multi-armed bandit.

If you want to learn which variation performs better, it must be possible for that metric to measure something in all of the variations within the MAB. To learn more, read Metrics and flag variations.

Build multi-armed bandits

You can view all of the MABs in your environment on the Multi-armed bandits list.

To build a MAB:

  1. Click Create, then choose Multi-armed bandits. A “Create multi-armed bandit” dialog appears.
  2. Enter a Name.
  3. Enter a Goal.
  4. Click Create multi-armed bandit. The Design tab opens.
  5. Choose a context kind to Randomize by.
  6. Select a Metric.
    • Hover over a metric to see which environments have received events for it. If no environments are receiving events, check that your SDKs are configured correctly.
    • Click Create to create and use a new metric.
  7. Choose a Flag or config.
    • Click Create flag or Create AgentControl config to create and use a new flag or AgentControl config.
  8. Choose a targeting rule for the Experiment audience.
    • If you want to restrict your MAB audience to only contexts with certain attributes, create a targeting rule on the flag or AgentControl config you include in the MAB and run the experiment on that rule. The context kind that the flag rule targets should match the randomization unit of your MAB.
    • If the flag or AgentControl config doesn’t have any targeting rules, the default rule is the only option.

The "Audience targeting" section with the default rule chosen.

The "Audience targeting" section with the default rule chosen.
  1. (Optional) If you want to exclude contexts in this MAB from certain other MABs and experiments, click Add experiment to exclusion layer and select a layer.
Expand layer options

A layer is a set of experiments and MABs that cannot share traffic with each other. All of the experiments and MABs within a layer are mutually exclusive, which means that if a context is included in one, LaunchDarkly will exclude it from any other experiments and MABs in the same layer.

To add the MAB to an existing layer:

  1. Click Select layer.
  2. Search for and choose the layer you want to add the MAB to.
  3. Enter a Reservation amount. This is the percentage of the contexts within this layer you want LaunchDarkly to include in this MAB.
  4. Click Save layer.

If you need to create a new layer:

  1. Click Create layer.
  2. Add a Name and Description.
  3. Click Create layer.
  4. Enter a Reservation amount. This is the percentage of the contexts within this layer you want LaunchDarkly to include in this MAB.
  5. Click Save layer.
  1. Select a Variation served to users outside this experiment.
  2. Select the Sample size for the MAB. This is the percentage of all of the contexts that match the MAB’s targeting rule that you want to include in the MAB.
  3. Select your desired Update frequency. This determines how often the MAB shifts traffic to the leading variation.
  4. By default, starting traffic is split equally between all of the included variations. If you want to exclude a variation from the MAB:
    • Click Edit next to “Variations split.”
    • For any variation you want to exclude, change the percentage to 0%.
    • Click Save audience split.
  5. (Optional) If you want to include the MAB in a holdout, select a Holdout name.
Multi-armed bandits cannot be in a holdout and in a layer at the same time

Multi-armed bandits can either be in a holdout or in a layer, but not both. If you added the multi-armed bandit to a layer, you will not see the option to add it to a holdout.

  1. Scroll to the top of the page and click Save.

If needed, you can save your in-progress MAB design to finish later. To save your design, click Save at the top of the creation screen. Your in-progress MAB design is saved and appears on the Multi-armed bandits list. To finish building the MAB, click on the MAB’s name and continue editing.

After you have created your MAB, the next step is to toggle on the flag. AgentControl configs are on by default. Then, you can start an iteration.

You can also use the REST API: Create experiment

Turn on flags or AgentControl configs

For a MAB to begin recording data, the flag or AgentControl config used in the MAB must be on. Targeting rules for AgentControl configs are on by default. To learn how to turn targeting rules on for flags, read Turning flags on and off.

Start iterations

After you create a MAB and toggle on the flag, you can start a MAB iteration in one or more environments. AgentControl configs are on by default.

To start a MAB iteration:

  1. Navigate to the Multi-armed bandits list.
  2. Click on the environment section containing the MAB you want to start.
    • If the environment you need isn’t visible, click the + next to the list of environment sections. Search for the environment you want, and select it from the list.

The environment selection menu.

The environment selection menu.
  1. Click on the name of the MAB you want to start an iteration for. The Design tab appears.
  2. Click Start.
  3. Repeat steps 1-4 for each environment you want to start an iteration in.

A multi-armed bandit with the "Start" button called out.

A multi-armed bandit with the "Start" button called out.

You can run a MAB indefinitely, allowing it to reallocate traffic as needed. If you are running a MAB on a time-boxed feature, such as a holiday promotion, then you can stop the MAB iteration when the promotion is over.

When you start an iteration of a MAB, LaunchDarkly sends the maintainer and anyone following the MAB an email, an in-app notification, and, if you have the Slack app integration configured, a Slack notification.

To learn more about starting and stopping iterations, read Starting and stopping experiment iterations.

You can also use the REST API: Create iteration