Filtering experiment results

This topic explains how to use attribute filters to narrow experiment results by attribute. Attribute filters allow you to analyze your experiment results by certain cross-sections of your experiment audience using context attributes. For example, if you wanted to view only results from users within Canada, you might narrow your results by users with a “Country” attribute value of Canada.

​

Add attribute filters

To filter results by attribute, you can add up to 20 attributes in the “Advanced” section when you create an experiment. To learn how, read Creating experiments. You can then filter your results by the different values of one of those attributes. You can also add attributes to an existing experiment from its Results tab:

1. Navigate to your experiment’s Results tab.
2. Scroll to the results table and click Add.
3. Select Attributes. An “Add attributes” dialog appears, listing the context attributes you can add.

4. Select the attributes you want to filter by. You can also type the name of an attribute that LaunchDarkly has not observed yet.
5. Click Add.

You can now filter your experiment results by these attributes.

​

View results filtered by attribute

To view experiment results filtered by attribute:

1. Navigate to your experiment’s Results tab.
2. Above the results table, click All attributes. A menu of attributes appears.
3. Hover over an attribute from the menu. A list of attribute values appears.
4. Check the box next to any attribute value you want to narrow your results by.

The graph updates to display results only from contexts that have the attribute values you selected:

​

Statistical adjustments

LaunchDarkly does not apply covariate adjustment (CUPED), multiple comparison correction (MCC), or sequential testing to filtered attribute results, even if LaunchDarkly applied those adjustments to the unfiltered results. We recommend using filtered results for exploration and hypothesis generation rather than final decision-making.

​

Attribute value update timing

After you add attributes to an existing experiment, the time it takes for LaunchDarkly to update the attribute values depends on the metric type and age of the experiment iteration:

Metric type	Iteration age	Time to update
LaunchDarkly hosted	Less than one day	Approximately five minutes
LaunchDarkly hosted	1-60 days	Approximately one hour
LaunchDarkly hosted	More than 60 days	Approximately one day
Warehouse native	Any	Every 15 minutes during the first 24 hours of an iteration, then at the warehouse integration’s refresh frequency

​

Attribute eligibility

You can add any attribute from your context data to an experiment, either when you create the experiment or while it is running. An attribute’s results display as empty if LaunchDarkly has not recorded values for it yet, or if it has too many unique values for LaunchDarkly to calculate results, as described in High-cardinality attributes.

​

High-cardinality attributes

For experiment iterations that use LaunchDarkly hosted metrics, LaunchDarkly does not calculate results for an attribute that has more than 500 unique values in the iteration’s audience. This typically applies to identifiers such as email addresses, user IDs, or names, where each value covers too few contexts to produce meaningful results. You can still add these attributes to an experiment, but their results do not appear. This limit does not apply to:

• Warehouse native experiment iterations, which calculate results for all attributes you add, regardless of how many unique values they have.
• Older hosted iterations that still use eligibility thresholds, as described in Eligibility for older experiment iterations.

​

Eligibility for older experiment iterations

Some older experiment iterations that use LaunchDarkly hosted metrics still use eligibility thresholds. For these iterations, the “Add attributes” dialog lists each attribute as eligible or ineligible based on how often LaunchDarkly observes it in your experiment audience. An attribute is eligible when its most common value:

• appears for at least 50 contexts in the experiment, and
• accounts for at least 5% of the experiment audience.

Attributes below these thresholds are ineligible. If the iteration is new or has low traffic, the dialog may display “No attributes available yet.” Attributes become eligible as the iteration collects more exposures.

​

Attribute value limits

For each attribute that LaunchDarkly calculates results for, LaunchDarkly displays results for up to the 20 most frequent values, ranked by the number of contexts having each value. All remaining values are combined into a single “(other)” group. If an attribute has 20 or fewer values, LaunchDarkly displays all of them and no “(other)” group appears. For experiment iterations that use LaunchDarkly hosted metrics, this applies only to attributes with 500 or fewer unique values. LaunchDarkly does not calculate results for attributes with more than 500 unique values. These attributes display no results at all. To learn more, read High-cardinality attributes. Limiting the number of values keeps results readable for attributes with many distinct values, and it reduces the risk of false positives from comparing many small segments. The limit also bounds the size of the queries that LaunchDarkly runs to calculate results. LaunchDarkly calculates results for the “(other)” group by combining the contexts from all remaining values into a single group, then calculating the metric directly over those contexts. The result is the same as if the “(other)” group were one attribute value. For mean and ratio metrics, the values you see always add up to your experiment totals. For percentile metrics, the “(other)” result is a percentile of the combined group. Percentile results never add up across values, because percentiles cannot be combined after they are calculated. This applies to both named values and to the “(other)” group. LaunchDarkly recalculates the top 20 values each time it updates results, over the full duration of the experiment iteration. If your data changes and a different value enters the top 20, the displayed values update the next time results update.