Ratio metrics

This topic explains how to create LaunchDarkly ratio metrics, which measure the ratio of two separate aggregations from Snowflake warehouse data sources.

About ratio metrics

A ratio metric is a complex metric type that computes the ratio of two separate metric aggregations: one for the numerator component and one for the denominator component. Each aggregation component is a simple metric created from a warehouse data source. LaunchDarkly divides the total of the numerator aggregation by the total of the denominator aggregation to compute the final metric value.

A ratio metric helps you determine how flag variations affect the relationship between two measurements. You can use ratio metrics to address the following use cases, which simple metrics do not support:

• Clustered analysis for experiments: Ratio metrics let you measure events that are grouped by an analysis unit that is finer-grained than the randomization unit chosen for the experiment. For example, you could measure the number of session errors generated relative to the total number of prompts that users created. In this example, you randomize the experiment by user contexts and configure a user aggregation for the denominator component, then configure a session aggregation for the numerator.

• Conversion metrics normalized by a separate event: Simple metrics can measure conversion events normalized by users who participate in the experiment. Ratio metrics can measure conversion events normalized by users that generate some other event, configured in the denominator aggregation. For example, you could measure the number of users who clicked on a search result, relative to the number of users who initiated a search.

You can add ratio metrics only to LaunchDarkly experiments that use the frequentist statistical methodology.

LaunchDarkly uses the computed value of the ratio to compare the performance of flag evaluations during an experiment. Because a ratio metric can track two separate analysis units, LaunchDarkly uses the statistical delta method to account for covariance of the two units. This helps to ensure that experiment recommendations remain accurate, even when analysis units in the denominator perform differently for a given experiment arm. To learn more, read Statistical methodology for frequentist experiments.

Aggregation types for ratio metrics

You configure each aggregation in a ratio metric separately. The numerator and denominator can use the same or different aggregation types, and can even use data from different warehouse data sources. However, the selected data sources must include context key mappings for one or more shared context kinds.

For example, if the data source for the numerator contains context key mappings for the “account,” “user,” and “session” context kinds but the denominator includes mappings for “session” and “device” contexts, then the ratio metric can only use “session” as the analysis unit. To learn more about mapping context keys in a warehouse data source, read Create data sources.

Ratio metrics support the same aggregation types available for simple metrics, plus one additional aggregation type, count_distinct, that counts distinct values from a warehouse column you provide. The available aggregation types are:

• Count uses sum aggregation to measure the total number of times a context generates the metric event (conversion metric).
• Sum uses sum aggregation to measure the total of the numerical values provided with a context’s metric events (numeric metric).
• Average uses average aggregation to measure the average numerical value provided with a context’s metric events (numeric metric).
• Count distinct uses count_distinct aggregation to measure the total number of times a unique value appears in a warehouse column associated with a context’s metric events (numeric metric). This option ignores duplicate or null values in the column.

The count_distinct aggregation type is available only for ratio metrics. It provides the flexibility to measure a distinct signal independently of the metric event count or event value. When you choose Count distinct, you provide the name of a warehouse column associated with the metric event to use for measuring distinct values.

For all ratio metric aggregations that correspond to a numeric metric, LaunchDarkly uses imputation to assign a zero value to any units that do not produce a metric event. To learn more, read Units without events.

Prerequisites and limitations

You can create ratio metrics only from Snowflake warehouse native metrics, for use with warehouse native Experimentation. Configure your Snowflake warehouse integration and create a data source before you create new warehouse native metrics. To learn more, read Metric data sources.

You cannot create ratio metrics from LaunchDarkly-hosted metric events, OpenTelemetry traces, or from warehouse data sources other than Snowflake.

You can add ratio metrics only to experiments that use the frequentist statistical methodology. To learn more, read Analyzing experiments.

Create a ratio metric

To create a ratio metric:

1. Open the Data section and navigate to the Metrics list.

2. Click Create metric. The “Create metric” dialog appears.

3. Click Ratio in the “Select metric structure” section. The “Select metric structure” section appears only if you have configured warehouse native Experimentation and a Snowflake warehouse data source.

4. Configure metric options for the “Numerator” component:

   1. Choose an existing data source from the Select metric data source menu. The data sources you choose for the numerator and denominator components must have at least one context key mapping to the same context kind.
   2. Enter the Event key to use for the numerator component of the metric.
   3. Choose an aggregation type from the Aggregate as menu:
      • Count uses sum aggregation to measure the total number of times a context generates the metric event (conversion metric).
      • Sum uses sum aggregation to measure the total of the numerical values provided with a context’s metric events (numeric metric).
      • Average uses average aggregation to measure the average numerical value provided with a context’s metric events (numeric metric).
      • Count distinct uses count_distinct aggregation to measure the total number of times a unique value appears in a column associated with a context’s metric events (numeric metric). This option ignores duplicate or null values in the column.
   4. If you chose Count distinct, enter a column name to use for the measurement in the Count distinct on field.
   5. (Optional) Click + Add winsorization or + Add custom measurement window as needed to configure the aggregation. To learn more, read Metric winsorization or Metric measurement window.

5. Repeat the previous step to configure an aggregation for the “Denominator” component of the metric. The aggregation type or warehouse column does not need to match the configuration of the numerator. However, the data sources you choose for the numerator and denominator must have at least one shared context key mapping.

   Both metric windows must complete

   If you configure a metric window for the numerator and denominator, windows for both components must complete before LaunchDarkly includes the ratio metric result in an experiment. To learn more, read Interpreting metrics excluded from experiments.

   After you configure the numerator and denominator, the window displays a complete metric definition using the values you selected.

6. Use the Analysis unit menu to select the analysis units to use for the metric. The same analysis units apply to both the numerator and denominator components of the ratio metric.

7. Choose higher is better or lower is better to define the success criteria for the computed ratio.

8. Enter a metric Name.

9. (Optional) Add a Description.

10. (Optional) Add any Tags.

11. (Optional) Update the Maintainer.

12. Click Create.