For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Sign inTry it free
DocsGuidesSDKsIntegrationsAPI docsTutorialsFlagship blog
DocsGuidesSDKsIntegrationsAPI docsTutorialsFlagship blog
  • Get started
    • Overview
    • Onboarding
    • Get started
    • Launch Insights
    • LaunchDarkly architecture
    • LaunchDarkly vocabulary
  • AgentControl
    • AgentControl
    • Manage AgentControl
  • Feature flags
    • Create flags
    • Target with flags
    • Flag templates
    • Manage flags
    • Code references
    • Contexts
    • Segments
  • Releases
    • Releasing features with LaunchDarkly
    • Release policies
    • Percentage rollouts
    • Progressive rollouts
    • Guarded rollouts
    • Feature monitoring
    • Release pipelines
    • Engineering insights
    • Release management tools
    • Applications and app versions
    • Change history
    • Restoring previous flag versions
  • Observability
    • Observability
    • Session replay
    • Error monitoring
    • Logs
    • Traces
    • Observability metrics
    • Product analytics events
    • LLM observability
    • Alerts
    • Dashboards
    • Service map
    • Vega for auto-remediation
    • Observability MCP server
    • Search specification
    • Observability settings
    • Observability integrations
  • Experimentation
    • Experimentation
    • Experiment metric types
    • Experiment configuration
    • Managing experiments
    • Analyzing experiments
    • Multi-armed bandits
    • Holdouts
      • Creating holdouts
      • Managing holdouts
  • Metrics and events
    • Metrics in LaunchDarkly
    • Creating metrics
    • Metric groups
    • Events
    • Autogenerated metrics
  • Warehouse native
    • Warehouse native metrics
    • Setting up external warehouses
    • Creating experiments using warehouse native metrics
  • Infrastructure
    • Connect apps and services to LaunchDarkly
    • LaunchDarkly in China and Pakistan
    • LaunchDarkly in the European Union (EU)
    • LaunchDarkly in federal environments
    • Public IP list
  • Your account
    • Projects
    • Views
    • Environments
    • Tags
    • Teams
    • Members
    • Roles
    • Account security
    • Feature previews
    • Billing and usage
    • Changelog
Sign inTry it free
LogoLogo
On this page
  • Manage holdouts
  • Read results
ExperimentationHoldouts

Managing holdouts

Was this page helpful?
Previous

Metrics in LaunchDarkly

Next
Built with

This topic explains how to manage holdouts and interpret their results.

Manage holdouts

You can view all of a project’s holdouts on the Holdouts list:

The Holdouts list.

The Holdouts list.

Holdouts can have four possible states:

  • Created: You have created the holdout, but have not yet added experiments or multi-armed bandits (MABs) to it.
  • Ready: You have added at least one experiment or MAB to the holdout, but none of the experiments or MABs have started.
  • Analyzing: At least one experiment or MAB is in the holdout, and it is currently running or it has one or more completed iterations.
  • Complete: The holdout is finished analyzing the holdout audience against the experiments’ or MABs’ audiences, and everyone is now receiving the same experience.

Click on the name of a holdout to view its details page.

The details page lists the holdout’s name, description, holdout amount, randomization unit, and any experiments or MABs included in the holdout:

The details page for a holdout.

The details page for a holdout.

You can also use the REST API: Get all holdouts

Read results

On the holdout details page, the results section displays the results of the metric for two variations:

  • “Not in holdout” includes any contexts included in an experiment or MAB within the holdout.
  • “In holdout” includes the contexts that were excluded from experiments or MABs within the holdout.

You can see which variation is performing better if at least one of the experiments or MABs in the holdout is running. However, we don’t recommend making any decisions about your holdout until all of the experiments or MABs within the holdout are finished and you have shipped winning variations for all of them. To learn how, read Analyzing experiments.

A holdout including experiments that are still running.

A holdout including experiments that are still running.

When you have recorded enough data, you can stop the holdout and analyze its results. To do this, navigate to the holdout details page and click End. The contexts that were in the holdout will no longer be excluded from future experiments or MABs.

You can then make a decision about the results of your holdout based on which variation is performing better:

  • If the “Not in holdout” variation is performing better, then this means your experiments or MABs are overall having a positive impact on the metric.
  • If the “In holdout” variation is performing better, then this means your experiments or MABs are overall having a negative impact on the metric, and you may want to examine the experiments or MABs you’re running and the variations you’re testing to figure out how you can build better or more effective experiments or MABs.

You can also use the REST API: Get holdout