Covariate adjustment and CUPED methodology

This guide explains the methodology and usage of CUPED (Controlled experiments Using Pre-Experiment Data) for covariate adjustment in LaunchDarkly Experimentation results.

Covariate adjustment refers to the use of variables unaffected by treatment, known as covariates, for:

• Variance reduction: Reduces the variance of experiment lift estimates, which increases measurement precision and experiment velocity.
• Bias removal: Removes the conditional bias of experiment lift estimates, which increases measurement accuracy.

In the context of online experimentation, Deng et al. (2013) introduced CUPED (short for Controlled Experiments Using Pre-Experiment Data), which can be thought of as a special case of the well-known Analysis of Covariance (ANCOVA) model with the pre-period version of the modeled outcome as a single covariate.

In this guide, we use the terms covariate adjustment, analysis of covariance (ANCOVA), and CUPED interchangeably.

Context

In a randomized experiment, there are three types of variables defined for each experiment unit, such as “user,” in a user-randomized experiment:

• Treatment: a variable indicating treatment for the unit. For example: 1 if the unit is assigned to the “treatment” variation, and 0 if assigned to the “control” variation.
• Outcomes: post-treatment variables that we want to measure experiment performance on, such as experiment revenue.
• Covariates: pre-treatment variables that we use to improve our measurement of the outcomes, typically for segmentation and variance reduction, such as pre-experiment revenue.

Outcomes are post-treatment variables. These are variables potentially affected by the treatment, or measured after the treatment is assigned. An example is revenue measured after the user enters the experiment.

Covariates must be pre-treatment variables, which are variables measured before the treatment is assigned, or variables unaffected by treatment. Examples include revenue measured before a user enters the experiment, which is measured before treatment, and gender, which is unaffected by the treatment.

Method

The goal of covariate adjustment is to improve the measurement of an experiment outcome, such as experiment revenue, through the use of prognostic covariates. Prognostic covariates are covariates predictive of the outcome. Pre-experiment revenue is an example of a prognostic covariate, which is typically predictive of experiment revenue. The ANCOVA model, and CUPED in particular, does this by leveraging the correlation, which is the strength of linear relationship, between an outcome and a set of covariates, with the goal of improving measurement precision and accuracy.

Variance reduction

We illustrate this with a simple example of an outcome $Y$, such as experiment revenue, and a covariate $X$, such as pre-experiment revenue.

In this example, there is a strong linear relationship between them for both treatment and control variations, shown in the scatter plot on the left below:

Predicting the observations in the treatment and control variations with, respectively, the sample means $\bar{Y}_T$ and $\bar{Y}_C$ results in a large variance for the errors, as illustrated in the plot on the right above.

However, we can leverage the linear relationship between $Y$ and $X$ by predicting the observations in the treatment and control variations with, respectively, the regression predictions $\hat{Y}_T = \hat{\alpha}_T + \hat{\beta}_T X$ and $\hat{Y}_C = \hat{\alpha}_C + \hat{\beta}_C X$, as shown in the scatter plot on the left below:

This results in smaller variance for the errors, as shown in the density plot on the right above. The above two scatter plots were inspired by those shown in Huitema (2011).

The correlation, which is the strength of the linear relationship between the outcome $Y$ and the covariate $X$, determines how much the error variance is reduced. The larger the correlation, the larger the variance reduction.

Specifically, if we denote the original error variance estimates for the two variations by, respectively, $s_T^2$ and $s_C^2$, and the new error variance estimates using CUPED by, respectively, $s_{T, CUPED}^2$ and $s_{C, CUPED}^2$, and the outcome-covariate correlations by, respectively, $r_T$ and $r_C$, then the following holds approximately:

$$\begin{aligned} s_{T, CUPED}^2 &\approx s_T^2 \left(1 - r_T^2 \right) \\ s_{C, CUPED}^2 &\approx s_C^2 \left(1 - r_C^2 \right) \end{aligned}$$

The proportional reduction in error variance is approximately the square of the correlations:

$$\begin{aligned} 1 - \frac{s_{T, CUPED}^2}{s_T^2} &\approx r_T^2 \\ 1 - \frac{s_{C, CUPED}^2}{s_C^2} &\approx r_C^2 \end{aligned}$$

If the correlations in both variations are $60\%$, the error variance will be reduced by $60\%^2 = 36\%$, and if they are $70\%$, the error variance will be reduced by $70\%^2 = 49\%$. The proportional reduction in the error variance translates to about the same proportion reduction in the variance of the experiment lift estimate, which translates to the same proportional reduction in experiment duration on average. Therefore, when the correlations are $60\%$, the experiment duration will be reduced by as much as $36\%$ on average, and when they are $70\%$, the experiment duration will be reduced by as much as $49\%$ on average. In other words, this can cut experiment duration nearly in half.

Bias removal

In addition to reducing the variance of lift estimates, CUPED applies an adjustment to the sample means $\bar{Y}_T$ and $\bar{Y}_C$ to produce the following covariate-adjusted means:

$$\begin{aligned} \bar{Y}_{T, CUPED} &= \bar{Y}_T - \hat{\beta_T} \left(\bar{X}_T - \bar{X}_{all} \right) \\ \bar{Y}_{C, CUPED} &= \bar{Y}_C - \hat{\beta_C} \left(\bar{X}_C - \bar{X}_{all} \right) \end{aligned}$$

Where $\bar{X}_T$ and $\bar{X}_C$ denote the covariate means for, respectively, the treatment and control variations, and $\bar{X}_{all}$ denote the covariate mean over all experiment variations. Although the unadjusted means $\bar{Y}_T$ and $\bar{Y}_C$ are unbiased estimators of the variation averages over many realizations of the experiment, for a specific experiment there could be some conditional bias. Conditional bias may occur due to the random imbalances between the treatment and control variation covariate means $\bar{X}_T$ and $\bar{X}_C$. As long as the linear regression model is correct, the adjustments $- \hat{\beta_T} \left(\bar{X}_T - \bar{X}_{all} \right)$ and $- \hat{\beta_C} \left(\bar{X}_C - \bar{X}_{all} \right)$ control for these imbalances and remove the conditional bias.

Implementation

In this section we discuss the scope and model for the CUPED implementation in the LaunchDarkly Experimentation feature.

CUPED availability

CUPED is available for experiments when the following criteria have been met:

• Average (mean) metrics: CUPED can be applied to metrics that use the “average” analysis method, including conversion metrics and continuous numeric metrics. CUPED is not applied to metrics that use a percentile analysis method.
• Metrics with historical data: CUPED can only be applied to a metric that has received events within the seven days prior to the start of the experiment. This is because CUPED requires historical metric data to compare current metric data to.
• Unsliced results: CUPED cannot be applied when you filter results by attribute.

Model

LaunchDarkly uses a general ANCOVA model allowing for unequal covariate slopes and unequal error variances by experiment group. LaunchDarkly’s model uses only one covariate: the pre-period version of the modeled outcome, which is the covariate originally proposed by Deng et al. (2013) for CUPED.

For a two-variation experiment, the model looks like this:

$$\begin{aligned} Y_i &= \alpha_A + \beta_A X_i + \epsilon_i \\ \epsilon_i &\sim Normal\left(0, \sigma_A^2 \right) \end{aligned}$$

where $A = T$ if unit $i$ is in the treatment variation and $A = C$ if unit $i$ is in the control variation. This model is fit as separate regressions for each variation.

The pre-period covariate is measured over a seven-day lookback window starting at the time of first exposure for a unit. Precedent for using only seven days is established by the implementation of the PrePost model for covariate adjustment for YouTube experiments, mentioned in Soriano (2019), which is the basis for our implementation.

There is also a tradeoff between using shorter versus longer windows in terms of relevance versus sufficiency. Shorter windows may have more relevance due to the recency of the information measured, but may not have captured all the information to optimize the outcome-covariate correlation. Longer windows capture more information, but risk including irrelevant information from older events, which may decrease the outcome-covariate correlation.

CUPED on the results tab

Each LaunchDarkly experiment indicates whether CUPED is off, applied, or not applied on its Results tab above the “Exposures” graph.

The CUPED statuses indicate:

• CUPED off: You have not enabled CUPED for this experiment.
• CUPED applied: You have enabled CUPED for this experiment and LaunchDarkly has applied it to at least one metric.
• CUPED not applied: You have enabled CUPED for this experiment but there is either no pre-experiment data, or there are no eligible metrics.

Advanced topics

For those interested, we cover some advanced topics in the following sections.

Causal inference interpretation

In a comparative study, whether a randomized experiment or an observational study, the goal is to perform causal inference, which includes estimating the causal effect of a treatment, for example, the causal effect of a new product feature on revenue.

Under the Neyman-Rubin potential outcomes framework for causal inference, we begin with individual potential outcomes (IPOs) $Y_i\left(1 \right)$ and $Y_i\left(0 \right)$ for, respectively, receiving the treatment and not receiving the treatment, for each individual $i$. The individual treatment effect (ITE) for individual $i$ is given by:

$$\begin{aligned} ITE_i &= Y_i\left(1 \right) - Y_i\left(0 \right) \end{aligned}$$

One estimand for the causal effect of treatment is the average treatment effect (ATE), which is the average of the ITEs:

$$\begin{aligned} ATE &= E \left( ITE_i \right) \\ &= E \left( Y_i\left(1 \right) - Y_i\left(0 \right) \right) \\ &= E \left( Y_i\left(1 \right) \right) - E \left( Y_i\left(0 \right) \right) \end{aligned}$$

This is the difference between the average potential outcomes (APOs) $E \left( Y_i\left(1 \right) \right)$ and $E \left( Y_i\left(0 \right) \right)$ of receiving and not receiving the treatment, respectively. An alternate causal estimand is the relative average treatment effect (RATE):

$$\begin{aligned} RATE &= \frac{E \left( Y_i\left(1 \right) \right)}{E \left( Y_i\left(0 \right) \right)} - 1 \end{aligned}$$

In the LaunchDarkly Experimentation feature, we estimate the APO $E \left( Y_i\left(a \right) \right)$ for each experiment variation $a$ for every combination of analysis time, experiment iteration, metric, and attribute. We then perform causal inference based on estimating the RATE for each treatment variation versus control.

Covariate-adjusted means

To perform causal inference, we first estimate the IPOs by their respective linear regression predictions for the treatment and control variations using the ANCOVA 3 model described earlier:

$$\begin{aligned} Y_i \left(1 \right) &= \hat{Y}_{T, i} = \hat{\alpha}_T + \hat{\beta}_T X_i \\ Y_i \left(0 \right) &= \hat{Y}_{C, i} = \hat{\alpha}_C + \hat{\beta}_C X_i \end{aligned}$$

The APOs are estimated by averaging the IPOs over all available units. In this case, the units are in both the treatment and control variations:

$$\begin{aligned} \hat{E} \left(Y_i \left(1 \right)\right) &= \frac{1}{n} \sum_{i=1}^n Y_i \left(1 \right) = \frac{1}{n} \sum_{i=1}^n \left( \hat{\alpha}_T + \hat{\beta}_T X_i \right) = \hat{\alpha}_T + \hat{\beta}_T \bar{X}_{all} \\ \hat{E} \left(Y_i \left(0 \right)\right) &= \frac{1}{n} \sum_{i=1}^n Y_i \left(0 \right) = \frac{1}{n} \sum_{i=1}^n \left( \hat{\alpha}_C + \hat{\beta}_C X_i \right) = \hat{\alpha}_C + \hat{\beta}_C \bar{X}_{all} \end{aligned}$$

where $\bar{X}$ denotes the average of the covariate over all units in both variations. Because the linear regression models have only one predictor, the estimated regression intercepts are given by:

$$\begin{aligned} \hat{\alpha}_T &= \bar Y_T - \hat\beta_T\bar{X}_T \\ \hat{\alpha}_C &= \bar Y_C - \hat\beta_C\bar{X}_C \\ \end{aligned}$$

Therefore, the estimated APOs are given by:

$$\begin{aligned} \hat{E} \left(Y_i \left(1 \right)\right) &= \bar{Y}_T - \hat{\beta}_T \bar{X}_T + \hat{\beta}_T \bar{X}_{all} = \bar{Y}_T - \hat{\beta}_T \left( \bar{X}_T - \bar{X}_{all} \right) = \bar{Y}_{T, adj} \\ \hat{E} \left(Y_i \left(0 \right)\right) &= \bar{Y}_C - \hat{\beta}_C \bar{X}_C + \hat{\beta}_C \bar{X}_{all} = \bar{Y}_C - \hat{\beta}_C \left( \bar{X}_C - \bar{X}_{all} \right) = \bar{Y}_{C, adj} \end{aligned}$$

We refer to $\bar{Y}_{T, adj}$ and $\bar{Y}_{C, adj}$ as covariate-adjusted means. They are the unadjusted sample means $\bar{Y}_{T}$ and $\bar{Y}_{C}$, minus the adjustments $\hat{\beta}_T \left( \bar{X}_T - \bar{X}_{all} \right)$ and $\hat{\beta}_C \left( \bar{X}_C - \bar{X}_{all} \right)$. This removes conditional bias due to the randomized imbalances between the covariate means $\bar{X}_T$ and $\bar{X}_C$ for both the treatment and control variations, respectively.

You can compute the estimated regression slopes with the following formulas:

$$\begin{aligned} \hat{\beta}_T &= r_T \frac{s_{Y, T}}{s_{X, T}} \\ \hat{\beta}_C &= r_C \frac{s_{Y, C}}{s_{X, C}} \end{aligned}$$

where:

• $s_{Y, T}$ and $s_{Y, C}$ are the sample standard deviation (SD) for the outcome in the treatment and control variations, respectively
• $s_{X, T}$ and $s_{X, C}$ are the sample SD for the covariate in the treatment and control variations, respectively, and
• $r_T$ and $r_C$ are the outcome-covariate correlation in the treatment and control variations, respectively.

We can show that the estimated SEs for the covariate-adjusted means for both the treatment and control variations are:

$$\begin{aligned} \hat{SE} \left(\bar{Y}_{T, adj} \right) &= s_{Y, T} \sqrt{1 - r_T^2}\sqrt{\frac{n_T - 1}{n_T - 2}}\sqrt{\frac{1}{n_T} + \frac{ (\bar{X}_T - \bar{X}_{all})^2 }{\left(n_T - 1 \right)s_{X, T}^2} } \\ \hat{SE} \left(\bar{Y}_{C, adj} \right) &= s_{C, T} \sqrt{1 - r_C^2}\sqrt{\frac{n_C - 1}{n_C - 2}}\sqrt{\frac{1}{n_C} + \frac{(\bar{X}_C - \bar{X}_{all})^2 }{\left(n_C - 1 \right)s_{X, C}^2} } \end{aligned}$$

where $n_T$ and $n_C$ are the sample sizes for the treatment and control variations, respectively.

When the sample sizes $n_T$ and $n_C$ are large and the imbalances $\bar{X}_T - \bar{X}_{all}$ and $\bar{X}_C - \bar{X}_{all}$ are negligible, the above SEs reduce to the following:

$$\begin{aligned} \hat{SE} \left(\bar{Y}_{T, adj} \right) &\approx s_{Y, T}\sqrt{\frac{1}{n_T}}\sqrt{1 - r_T^2} = \hat{SE} \left(\bar{Y}_T \right) \sqrt{1 - r_T^2} \\ \hat{SE} \left(\bar{Y}_{C, adj} \right) &\approx s_{Y, C}\sqrt{\frac{1}{n_C}}\sqrt{1 - r_C^2} = \hat{SE} \left(\bar{Y}_C \right) \sqrt{1 - r_C^2} \end{aligned}$$

Therefore, the proportional variance reduction for each is approximately equal to the squared correlation for the variation, as we showed earlier:

$$\begin{aligned} 1 - \frac{\hat{Var} \left(\bar{Y}_{T, adj} \right)}{\hat{Var} \left(\bar{Y}_T \right)} &\approx r_T^2 \\ 1 - \frac{\hat{Var} \left(\bar{Y}_{C, adj} \right)}{\hat{Var} \left(\bar{Y}_C \right)} &\approx r_C^2 \end{aligned}$$

Frequentist and Bayesian approaches

For frequentist estimates, the estimates of the APOs are the above covariate-adjusted means $\bar{Y}_{T, adj}$ and $\bar{Y}_{C, adj}$. In the Bayesian model, the APO estimates are regularized using empirical Bayes priors. To learn more, read Statistical methodology for Bayesian experiments and Statistical methodology for frequentist experiments.

The Bayesian results without covariate adjustment through CUPED continue to use the normal-normal model for custom conversion count and custom numeric continuous metrics and the beta-binomial model for custom conversion binary, clicked or tapped, and page viewed metrics. However, the Bayesian results with covariate adjustment through CUPED will use the normal-normal model for all metrics using the “average” analysis method, including custom conversion binary metrics. Under this model, we assume the following prior distribution for the parameter estimated in variation $A_{T, C}$ :

$$\begin{aligned} \theta_{prior, A} \sim Normal\left(\mu_{0, A}, \sigma_{0, A}^2 \right) \end{aligned}$$

For details on the prior mean $\mu_{0, A}$ and $\sigma_{0, A}^2$, read Statistical methodology for Bayesian experiments.

LaunchDarkly provides a frequentist estimate $\hat\theta_A$ and its estimated standard error $\hat{SE}\left(\hat\theta_A \right)$. For the non-CUPED results, the estimate is the sample mean. For CUPED results, the estimate in the covariate-adjusted mean $\hat\theta_A = \bar{Y}_{A, adj}$, with details provided in the previous section.

We define precision as the inverse of the variance, which is equivalent to the inverse of the squared standard error. Therefore, the estimated precisions of the prior distribution and the frequentist estimate are, respectively:

$$\begin{aligned} Prec\left(\theta_{prior, A} \right) &= \frac{1}{\sigma_{0, A}^2} \\ Prec\left(\hat\theta_A \right) &= \frac{1}{\hat{SE}\left(\hat\theta_A \right)^2} \end{aligned}$$

Define the following precision sum and weight:

$$\begin{aligned} S &= Prec\left(\hat\theta_A \right) + Prec\left(\theta_{prior, A} \right) \\ w &= \frac{Prec\left(\hat\theta_A \right)}{S} \end{aligned}$$

Then the posterior distribution of the estimated parameter is given by:

$$\begin{aligned} \theta_{posterior, A} \sim Normal \left(w \hat{\theta} + \left(1 - w\right)\mu_{0, A}, S^{-1} \right) \end{aligned}$$

where the posterior mean is given by the precision-weighted average of the frequentist estimate $\hat{\theta}$ and the prior mean $\mu_{0, A}$, and the posterior variance is the inverse of the sum of the frequentist estimate precision $Prec\left(\hat\theta_A \right)$ and the prior precision $Prec\left(\theta_{prior, A} \right)$.

References

Deng, Alex, Ya Xu, Ron Kohavi, and Toby Walker (2013). “Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data.” WSDM’13, Rome, Italy.

Fisher, Ronald A. (1932). Statistical Methods for Research Workers. Oliver and Boyd. Edinburgh, 4th ed.

Huitema, Bradley (2011). Analysis of Covariance and Alternatives: Statistical Methods for Experiments, Quasi-Experiments, and Single-Case Studies, 2nd ed. Wiley.

Lin, Winston (2013). “Agnostic Notes on Regression Adjustments to Experimental Data: Reexamining Freedman’s Critique.” Annals of Applied Statistics, 7(1): 295-318.

Soriano, Jacopo (2019). “Percent Change Estimation in Large Scale Online Experiments.” https://arxiv.org/pdf/1711.00562.pdf.

Yang, Li and Anastasios A. Tsiatis. (2001). “Efficiency Study of Estimators for a Treatment Effect in a Pretest-posttest Trial.” American Statistician, 55: 314-321.

Ye, Ting, Jun Shao, Yanyao Yi, and Qingyuan Zhao (2023). “Toward Better Practice of Covariate Adjustment in Analyzing Randomized Clinical Trials.” Journal of the American Statistical Association, 118(544): 2370-2382.