Analyzing experiments

Overview

This topic explains the experiment analysis options available in LaunchDarkly.

Bayesian and frequentist analysis

LaunchDarkly offers two different approaches for analyzing your experiment: Bayesian and frequentist. Within the frequentist approach, you can choose a traditional fixed-horizon analysis or a sequential analysis.

To learn more, read the following guides:

• Choosing a statistical methodology
• Statistical methodology for Bayesian experiments
• Statistical methodology for frequentist experiments

CUPED

Controlled-experiment using pre-experiment data (CUPED) is a statistical method that uses data collected before an experiment begins to reduce variance in experiment results. Applying CUPED to experiment results makes it easier to detect differences between variations.

To learn more, read Covariate adjustment and CUPED methodology.

Multiple comparisons correction

Multiple comparisons correction (MCC) helps reduce false positives in frequentist experiment results.

To learn more, read the following topics:

• To learn how to apply MCC to a new experiment, read Creating experiments.
• To learn how to remove MCC from an existing experiment, read Statistics information
• To learn about our MCC methodology, read Multiple comparisons correction.

Sequential testing

Sequential testing lets you make decisions about frequentist results without needing to calculate a sample size first.

To learn more, read the following topics:

• To learn how to apply sequential testing to a new experiment, read Creating experiments.
• To learn more about when to use sequential testing, read Fixed-horizon versus sequential.

Sample ratio mismatch (SRM) detection

In an experiment, the sample ratio is the ratio between end users in each of your experiment variations. An SRM is a mismatch between the number of users expected and the actual number of users included in each variation. SRM detection alerts you to any issues with your JavaScript-based SDK setups.

To learn more, read Sample ratio mismatch.

Stratified sampling

Stratified sampling helps eliminate covariate imbalance and reduce false positives and false negatives in your experiment results.

To learn more, read the following topics:

• To learn how to apply stratified sampling to a new experiment, read Creating experiments.
• To learn how it works, read Stratified sampling.

Related topics

To learn how to interpret the results of your experiments, read the following topics:

• Experiments results data
• Bayesian experiment results
• Frequentist experiment results.