Sample size calculations

This topic explains how to use the sample size calculator to estimate how many units your experiment needs and how long it needs to run to return meaningful results.

Sample size targets apply to frequentist, fixed-horizon experiments only. They do not apply to Bayesian experiments or to frequentist experiments that use sequential testing, because those methods let you act on results without waiting for a predetermined sample size. To learn more, read Fixed-horizon versus sequential.

Prerequisites

To calculate a sample size, you need the following:

• An experiment that uses a frequentist statistical approach with a fixed horizon. To learn more, read Creating experiments.
• An estimate of the current performance of your primary metric.
• An estimate of how much daily traffic matches your experiment’s targeting rule.

About the sample size calculator

The sample size calculator estimates two things from the inputs you provide:

• Sample size: The total number of units across all of your experiment’s variations that you need to detect the effect you care about.
• Duration: How long the experiment needs to run to reach that sample size, based on the traffic you expect.

When you save an estimate, LaunchDarkly stores it as the experiment’s sample size target, so you know when the experiment will be ready for a decision. Deciding on a sample size before you start the experiment is what keeps your false positive rate at the significance level you chose. To learn why, read Sample size calculations for frequentist experiments.

You can also use the sample size calculator on the LaunchDarkly website if you want to estimate a sample size outside of the LaunchDarkly user interface (UI). This calculator has the additional ability to estimate the minimum detectable effect (MDE) for a specified sample size.

The website calculator accepts a standard deviation ratio, which lets you assume the treatment varies more or less than the control. The in-app calculator always assumes a ratio of 1. If you set a ratio other than 1 on the website, the two calculators return different sample sizes.

Sample size calculator fields

The sample size calculator groups its fields into three sections: “Metric,” “Traffic,” and “Statistical settings.” Each section is explained below.

Metric

The “Metric” section defines the primary metric the experiment is tracking.

Choose a Metric type of Conversion or Numeric. The baseline fields change to match the metric type you choose.

You cannot calculate a sample size for a numeric metric that uses percentile analysis, because the calculator estimates sample sizes for metrics analyzed as means. To learn more about analysis methods, read Custom numeric metrics.

For a conversion metric, complete the following field:

• Baseline conversion rate: The current conversion rate for the control variation. For example, if 10% of the contexts that encounter your control variation convert today, enter 10%.
• Minimum detectable effect: The smallest relative change worth detecting, as a percentage of the baseline. Smaller values require more data, so choose the smallest change that would affect your decision-making.

For a numeric metric, complete the following fields:

• Baseline mean: The average metric value for the control variation. For example, if your control variation currently averages an order value of 100, enter 100.
• Standard deviation: How much the metric values vary across units. Metrics with larger variance require a larger sample size.
  • If you don’t know the standard deviation for your metric, you can estimate it from historical data for the same metric.
  • The calculator assumes the treatment variations have the same standard deviation as the control. You provide one standard deviation, and it applies to every variation.
  • If your primary metric is a ratio metric, or if the metric’s analysis units differ from the experiment’s randomization units, enter a standard deviation calculated at the randomization unit level. For example, you can use the Delta method to calculate the effective randomization unit-level standard deviation. The calculator returns a sample size in randomization units, so a standard deviation calculated at the analysis unit level returns an estimate that does not reflect the number of units your experiment needs.
• Minimum detectable effect: The smallest relative change worth detecting. Smaller values require substantially more data, so choose the smallest acceptable change.

Traffic

The “Traffic” section estimates how much traffic the experiment will receive. LaunchDarkly uses these values to convert the required sample size into a duration.

Complete the following fields:

• Estimated daily traffic: The total number of eligible units per day. Eligible units are the units that match your experiment’s targeting rule, whether or not they enter the experiment.
• Percent of users in this experiment: The share of eligible traffic that enters the experiment. This should match the audience allocation on your experiment. To learn more, read Allocating experiment audiences.

For example, an estimated daily traffic of 10,000 units with a traffic allocation of 50% means 5,000 units per day enter the experiment.

Statistical settings

The “Statistical settings” section determines the confidence and power for the analysis.

Complete the following fields:

• Significance level: The configured allowable rate of false positives for an experiment. 95% is the default, which corresponds to a significance level of 0.05. Higher values require a larger sample size.
• Statistical power: The probability that your experiment detects a difference of the size you specified as the minimum detectable effect, if that difference exists. 80% is the standard value. Higher values require a larger sample size.
• Direction of hypothesis test: Choose Two-sided when you are unsure whether the treatment will perform better or worse than the control. Choose One-sided when you are confident about the direction of the difference and only want to test for it in that direction.

Set these values to match the statistical settings on your experiment.

Estimates

The “Estimates” panel updates as you change your inputs. It displays:

• Sample size: The total number of units required across all of your experiment’s variations.
• Duration: The number of days and weeks needed to reach that sample size, based on the number of units per day your traffic inputs imply.

If the duration is longer than you can run your experiment for, you can make adjustments with certain tradeoffs:

• Increasing the MDE: This is a change in expectation. You look for larger, more impactful changes while keeping the mathematical integrity of your test intact. The risk is missing a smaller change.
• Increasing traffic allocation: This brings in data faster to shorten the timeline, but it exposes a broader audience to unproven variations, increasing the risk to user experience if a variant performs poorly.
• Lowering the statistical power: This directly increases the risk of missing a winner because the test lacked the power to see it.
• Increasing the significance level: This directly increases the risk of falsely concluding a winner when the difference was only random noise.

We recommend running experiments for at least one week, even when the calculator estimates a shorter duration. Running for at least a full week accounts for differences in end user behavior between weekdays and weekends.

Calculate sample sizes

To calculate a sample size:

1. Navigate to the experiment’s design.
2. Scroll to the “Sample size” section.
3. Click Attach calculation. The sample size calculator appears.
4. Complete the Metric, Traffic, and Statistical settings sections.
5. Review the estimated sample size and duration in the “Estimates” panel.
6. Click Save.

To discard your inputs and return every field to its default value, click Reset. To close the dialog without saving an estimate, click Cancel.