For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Sign inTry it free
DocsGuidesSDKsIntegrationsAPI docsTutorialsFlagship blog
DocsGuidesSDKsIntegrationsAPI docsTutorialsFlagship blog
  • Get started
    • Overview
    • Onboarding
    • Get started
    • Launch Insights
    • LaunchDarkly architecture
    • LaunchDarkly vocabulary
  • AgentControl
    • AgentControl
    • Manage AgentControl
  • Feature flags
    • Create flags
    • Target with flags
    • Flag templates
    • Manage flags
    • Code references
    • Contexts
    • Segments
  • Releases
    • Releasing features with LaunchDarkly
    • Release policies
    • Percentage rollouts
    • Progressive rollouts
    • Guarded rollouts
    • Feature monitoring
    • Release pipelines
    • Engineering insights
    • Release management tools
    • Applications and app versions
    • Change history
    • Restoring previous flag versions
  • Observability
    • Observability
    • Session replay
    • Error monitoring
    • Logs
    • Traces
    • Observability metrics
    • Product analytics events
    • LLM observability
    • Alerts
    • Dashboards
    • Service map
    • Vega for auto-remediation
    • Observability MCP server
    • Search specification
    • Observability settings
    • Observability integrations
  • Experimentation
    • Experimentation
    • Experiment metric types
    • Experiment configuration
    • Managing experiments
    • Analyzing experiments
      • Experiment results data
      • Bayesian experiment results
      • Frequentist experiment results
      • Filtering experiment results
    • Multi-armed bandits
    • Holdouts
  • Metrics and events
    • Metrics in LaunchDarkly
    • Creating metrics
    • Metric groups
    • Events
    • Autogenerated metrics
  • Warehouse native
    • Warehouse native metrics
    • Setting up external warehouses
    • Creating experiments using warehouse native metrics
  • Infrastructure
    • Connect apps and services to LaunchDarkly
    • LaunchDarkly in China and Pakistan
    • LaunchDarkly in the European Union (EU)
    • LaunchDarkly in federal environments
    • Public IP list
  • Your account
    • Projects
    • Views
    • Environments
    • Tags
    • Teams
    • Members
    • Roles
    • Account security
    • Feature previews
    • Billing and usage
    • Changelog
Sign inTry it free
LogoLogo
On this page
  • Overview
  • Results update frequency
  • Summary
  • Bayesian experiments
  • Exposures
  • Statistics information
  • Results table
  • Further analyzing results
  • Data Export
  • Qualitative user feedback
ExperimentationAnalyzing experiments

Experiment results data

Was this page helpful?
Previous

Bayesian experiment results

Next
Built with

Overview

This topic explains how to interpret an experiment’s results and apply its findings to your product.

The data an experiment has collected is represented in its Results tab. The Results tab provides information about each variation’s performance in the experiment, how it compares to the other variations, and which variations are likely to be best or beat the control out of all the tested options. Understanding how to read this tab can help you make informed decisions about when to edit, stop, or choose a winning variation for your experiments.

Here is an example Results tab for an experiment:

An experiment's results tab.

An experiment's results tab.

Results update frequency

How often LaunchDarkly updates experiment results depends on the age of the experiment:

  • In the first 24 hours of an experiment, LaunchDarkly updates the results at least every ten minutes.
  • For experiments that are between 1-60 days old, LaunchDarkly updates the results every hour.
  • For experiments more than 60 days old, LaunchDarkly updates the results once per day.

Summary

The “Summary” section displays the experiment’s hypothesis and key takeaways, including whether or not you have enough data to determine a winning variation for the experiment.

Bayesian experiments

For Bayesian experiments, the summary includes a sample size estimator that gives an estimate of how much more traffic needs to encounter your experiment before reaching your chosen probability to be best. To learn more, read Sample sizes for Bayesian experiments.

The Summary section of an experiment's results page.
The Summary section of an experiment's results page.

Exposures

The “Exposures” section displays the percentage of contexts exposed to each variation over time. To learn more, read Experiment sample size and run time.

The "Exposures" section.

The Exposures section.

Statistics information

The following statistics information displays above the results tables:

  • Significance level (frequentist experiments only): The range of values within which, if you repeated the experiment many times, would contain the true value of the relative difference between the treatment and the control. To learn more, read Frequentist experiment results.
  • Threshold (Bayesian experiments only): The threshold represents how confident you want to be in an experiment’s results before making a decision. The threshold determines the width of your credible intervals, and when LaunchDarkly declares a variation the winning variation. when You select your desired threshold when you create an experiment. To learn more, read Bayesian experiment results.
  • SRM: Specifies if LaunchDarkly has detected a sample ratio mismatch (SRM). SRMs indicate that there may be an issue with the implementation of your JavaScript-based SDKs. To learn more, read Sample ratio mismatch.
  • CUPED: Specifies if LaunchDarkly has enabled covariate adjustment (CUPED) for this experiment. CUPED uses data collected before an experiment began to reduce variance in experiment results.
  • MCC (frequentist experiments only): Specifies if you have multiple comparisons correction (MCC) enabled to help reduce false positives in your experiment.
    • To toggle off MCC on a running or completed experiment, click on the pencil icon above the “Exposures” section and uncheck the Multiple comparisons correction checkbox.
  • Sequential testing: (Frequentist experiments only) Whether or not you have enabled sequential testing to allow you to act on the results at any time.

An experiment's statistics information.

An experiment's statistics information.

Results table

An experiment’s results table displays each variation’s performance in the experiment. You can choose to view the results by forest plot, arm averages, probability density, or relative difference.

To learn more, read Bayesian experiment results and Frequentist experiment results.

A frequentist experiment's results table.

A frequentist experiment's results table.

The results table groups variation results by the metrics chosen for the experiment. The “Sample size” column shows the number of units the metric contributes to experiment results. Certain metric configuration options can cause units to be excluded from the experiment. Click a value in this column to view a breakdown of any units that were excluded from consideration. To learn more, read Metric measurement window.

Further analyzing results

You can further analyze your results using Data Export or by implementing qualitative user feedback.

Data Export

If you’re using Data Export, you can further analyze your experiment data using third-party tools. To learn more, read Data Export.

Qualitative user feedback

Experimentation provides useful quantitative feedback, but you may be interested in gathering qualitative feedback as well. To do this, you can configure your application to send user feedback to LaunchDarkly. To learn how, read Collecting user feedback.