Choosing a difference type for guarded rollouts
Overview
This topic explains how to choose between absolute difference and relative difference when configuring regression detection for a guarded rollout.
Absolute difference is the default difference type for guarded rollouts. LaunchDarkly recommends it for binary and sparse metrics, such as error counts and error rates near zero.
Relative difference is available for metrics where performance is evaluated based on proportional change between the treatment and the control.
Choosing a difference type
When you configure a guarded rollout, you select a difference type for each metric. You can select a different difference type for each metric in the same guarded rollout.
Choose absolute difference if:
- The metric is binary or sparse.
- The control value is near zero.
- You evaluate performance based on absolute change, such as the number of additional errors.
- You want stronger control of false positives for non-percentile metrics. Relative difference relies on an approximation method for variance estimation, which may introduce additional estimation error compared to absolute difference.
Choose relative difference if:
- The control value is stable and not near zero.
- You evaluate performance based on percentage change between the treatment and the control.
After a guarded rollout starts, you cannot change the difference type.
Absolute difference
Absolute difference measures the absolute change between the treatment value and the control value using the metric’s units.
LaunchDarkly calculates this value by subtracting the control value from the treatment value.
The result is expressed using the metric’s unit. If a metric is defined as a percentage, the difference reflects the change in that percentage value. If a metric is defined as a count or duration, the difference is expressed in counts or time units.
Relative difference
Relative difference measures proportional change between the treatment value and the control value.
LaunchDarkly calculates this value by subtracting the control value from the treatment value and dividing the result by the control value.
When the control value is very small or zero, this calculation can produce unstable results:
- Small changes in the metric value may appear as large percentage changes.
- A control value of zero results in an undefined relative difference calculation.
- Regression signals may not reflect meaningful changes in the underlying metric.
The result is expressed as percent change relative to the control value.
How difference type affects regression detection
In a guarded rollout, LaunchDarkly compares the performance of a new variation, called the treatment, to the original variation, called the control, to determine whether a regression has occurred.
The selected difference type determines how regression thresholds are evaluated and whether automated actions, such as rollback, are triggered.
For example, if the control error rate is 0% and the treatment error rate increases to 1%, relative difference cannot be calculated because the control value is zero. Absolute difference reflects an absolute change of 1 percentage point (pp) and can be evaluated directly.
If the control error rate is 5% and the treatment error rate increases to 6%, relative difference reflects a 20% increase, while absolute difference reflects an absolute change of 1 pp.
When you use absolute difference, regression thresholds are interpreted using the same unit as the metric. When you use relative difference, regression thresholds are interpreted as percent change relative to the control value.
For binary metrics, absolute difference is expressed in percentage points (pp), while relative difference is expressed as percent change (%). This formatting helps distinguish the selected difference type when reviewing results.
How difference type affects charts and results
The selected difference type determines how metric results are presented in the LaunchDarkly UI.
When you use absolute difference, charts display the absolute change using the metric’s unit. When you use relative difference, charts display percent change relative to the control value. Regression threshold lines and hover details reflect the selected difference type.
LaunchDarkly calculates and displays results only for the selected difference type. You cannot switch between absolute difference and relative difference views after the rollout has started.
Key limitations
The following constraints apply to difference types in guarded rollouts:
- You must select a difference type during guarded rollout setup. After the rollout starts, you cannot change the difference type.
- Difference type applies only to guarded rollouts.
- Selecting a difference type does not change how a metric is defined or calculated.
- Selecting a difference type does not change the underlying statistical model used to evaluate regressions.
- Experiments and progressive rollouts use different evaluation methods and do not use difference type.
