Analytic formulas for frequentist experiment variation means

Overview

LaunchDarkly displays analytic formulas for all experiment variation means. This guide explains the statistical methodology for these formulas for experiments using frequentist statistics, and how they are useful for validating your results.

Data mean

Whether an experiment’s results table displays the conversion rate or mean is dependent on the kind of metrics you’re using, and whether LaunchDarkly has applied covariate adjustment through CUPED. Experiments using conversion metrics display the conversion rate, and experiments using numeric metrics display the mean.

Formulas without CUPED applied

When CUPED is not applied to an experiment metric, the data mean equals the observed sample mean.

The formula is different for conversion metrics and numeric metrics:

• Conversion metrics, including custom conversion binary, custom conversion count, page viewed, and clicked or tapped metrics, use the total number of conversions divided by the total number of exposures: $DataMean = SampleMean = Conversions / Exposures$
• Numeric metrics use the total value divided by the total number of exposures: $DataMean = SampleMean = TotalValue / Exposures$

When you hover over the “Conversion rate” and “Mean” headings, you can view the above formulas for the data mean when LaunchDarkly does not apply CUPED:

Formulas with CUPED applied

When CUPED is applied to an experiment metric, the estimated data mean equals the sample mean minus an adjustment, where the sample mean is either conversions over exposures or total value over exposures:

where the adjustment is the product of the following components:

• Correlation: the correlation between the experiment outcome and the pre-period covariate in the CUPED model
• Standard deviation ratio: the ratio of the standard deviation (SD) of the experiment outcome to that of the pre-period covariate
• Covariate imbalance: the difference between the covariate mean for the experiment variation and the covariate mean for all variations, which is a random imbalance due to sampling variability

This adjustment explains why the estimated conversion rate for a conversion metric may not equal the number of conversions divided by the number of exposures, and why the estimated mean for a numeric metric may not equal the total value divided by the number of exposures.

When you hover over the “Conversion rate” and “Mean” headings, you can view the above formulas for the data mean when LaunchDarkly applies CUPED:

Conclusion

This guide explains the formulas LaunchDarkly uses to derive experiment variation means in frequentist experiments, and how they are useful for validating your results.

To learn more about the statistical methods we use in frequentist experiments, read Experimentation statistical methodology for frequentist experiments.

To learn how to interpret the experiment results page, read Frequentist experiment results.