Sending OpenTelemetry traces to LaunchDarkly

Overview

This topic explains how to send OpenTelemetry traces to LaunchDarkly. LaunchDarkly converts this data into events that LaunchDarkly metrics track over time. Experimentation and guarded rollouts use these metrics to measure flag performance.

This topic covers:

• how to send trace data to LaunchDarkly
• how to configure your collector to receive, process, and export telemetry data
• how LaunchDarkly automatically creates metrics from OpenTelemetry trace data

About OpenTelemetry

OpenTelemetry (OTel) is an open source observability framework and toolkit designed to create and manage telemetry data such as traces, metrics, and logs. You can use LaunchDarkly’s OpenTelemetry protocol (OTLP) endpoint to send OpenTelemetry traces to LaunchDarkly.

LaunchDarkly converts the information contained in these traces to produce events for use with Experimentation and guarded rollouts. Typically the information you collect in the traces is related to API performance and errors.

Because OpenTelemetry is vendor- and tool-agnostic, you can reuse the instrumentation that may already exist in your code to create LaunchDarkly metrics without changing application code.

LaunchDarkly supports receiving trace data over gRPC, HTTP/protobuf, and HTTP/JSON.

Send trace data to LaunchDarkly

To send OpenTelemetry trace data to LaunchDarkly, we recommend using the OpenTelemetry Collector. The OpenTelemetry Collector is a vendor-agnostic proxy that can receive, process, and export telemetry data. You can configure an OpenTelemetry pipeline with a processor to filter out data that is not relevant to LaunchDarkly, reducing data transport costs, and to send data to LaunchDarkly in addition to other OTLP backends that you may already be using.

LaunchDarkly processes two types of data from your OpenTelemetry traces:

• HTTP span attributes, including latency, 5xx occurrences, and other errors, where the span has or overlaps with another span that has at least one feature flag span event. This includes nested spans.
• Exception span events that occur after a feature flag span event on the same trace. If the exception occurs before the feature flag event, LaunchDarkly does not capture it.

A feature flag span event is defined as any span event that contains a feature_flag.context.key attribute. LaunchDarkly ignores traces that do not include span events with this attribute.

To ensure your spans have compatible feature flag events, configure your SDK to use the OpenTelemetry tracing hook.

In most SDKs, the OTel traces are specific to the LaunchDarkly project and environment that you specify in the Collector configuration described below. In the .NET (server-side) SDK, you can provide the LaunchDarkly client-side ID in the tracing hook. This enables you to send traces for multiple projects and environments using one Collector.

Try it in your SDK: OpenTelemetry

Generate a new LaunchDarkly service token

All trace data sent to LaunchDarkly must be authenticated with a LaunchDarkly personal or service API access token with a Writer role, or a custom role that allows the token to perform the importEventData action on the LaunchDarkly environment you are targeting.

Here’s how to create an API service token with the appropriate inline policy:

1. Click the gear icon in the left sidenav to view Organization settings. The General settings page appears.

2. Click Authorization.

3. Click Create token. The “Create access token” dialog appears.

4. Give your token a human-readable Name, such as “Compass integration.”

5. Give your token an “Inline policy” Role.

6. Click the Advanced editor button and specify the following policy in the Editor field to allow the token to perform the importEventData action on all LaunchDarkly environment resources:

   OpenTelemetry Import Inline policy

   1
   [
   2
     {
   3
       "resources": [
   4
         "proj/*:env/*"
   5
       ],
   6
       "actions": [
   7
         "importEventData"
   8
       ],
   9
       "effect": "allow"
   10
     }
   11
   ]

7. Select This is a service token.

8. Click Save token and copy the newly created token to your clipboard. You will need it to configure the OpenTelemetry collector in the following section.

Collector configuration

Collector pipelines consists of three components: receivers, exporters, and processors.

The exporter requires specific headers:

If you are using the .NET (server-side) SDK v8.6 or earlier, or any other SDK, then you must provide the X-LaunchDarkly-Project and X-LaunchDarkly-Environment headers. When you do, LaunchDarkly imports all traces into the environment specified by these headers. This is shown in the “single environment” examples, below. To import traces into multiple environments when using these headers, you must set up a separate OpenTelemetry Collector for each environment, and configure each of your services to send OTel traces to the correct Collector.

If you are using the .NET (server-side) SDK v8.7 or above, then you should omit the X-LaunchDarkly-Project and X-LaunchDarkly-Environment headers. In this version of the SDK, LaunchDarkly automatically imports each trace into the correct environment based on an environment field generated by the LaunchDarkly OTel Hook. If you send traces this way, you may set up a single OTel Collector that accepts traces from all of your services across environments. This is shown in the “multiple environment” examples, below.

To configure a pipeline to send trace data to LaunchDarkly, modify the Collector’s config file, otel-collector-config.yaml, as follows:

1
# The receivers specify how the Collector receives data.
2
# In this example, it receives data using the OpenTelemetry Protocol (OTLP) over gRPC and HTTP.
3
receivers:
4
  otlp:
5
    protocols:
6
      grpc:
7
      http:
8
9
# The exporters specify how the Collector sends data.
10
# In this example, it sends data to LaunchDarkly using HTTP.
11
exporters:
12
  otlphttp/launchdarkly:
13
    traces_endpoint: https://events.launchdarkly.com/otlp/traces
14
    headers:
15
      # The Authorization header must be a personal or service API access token with the `importEventData` action.
16
      # The project and environment are set automatically based on your SDK configuration.
17
      Authorization: ${env:LD_ACCESS_TOKEN}
18
19
# The processors specify how the Collector processes the trace data.
20
# In this example, it groups spans that belong to the same trace,
21
# and only passes along span events related to feature flags or exceptions.
22
processors:
23
24
  # The groupbytrace processor groups all spans that belong to the same trace together.
25
  # This is required to ensure LaunchDarkly receives a complete trace in a single request.
26
  groupbytrace:
27
    wait_duration: 10s
28
29
  # Remove all span events that are not "feature_flag" or "exception"
30
  filter/launchdarkly-spanevents:
31
    error_mode: ignore
32
    traces:
33
      spanevent:
34
        - 'not (name == "feature_flag" or name == "exception")'
35
36
  # Remove all spans that do not have an HTTP route or any span events remaining
37
  # after the previous filter has been applied
38
  filter/launchdarkly-spans:
39
    error_mode: ignore
40
    traces:
41
      span:
42
        - 'not (attributes["http.route"] != nil or Len(events) > 0)'
43
44
  batch:
45
46
extensions:
47
  health_check:
48
49
service:
50
  pipelines:
51
    # Add a new pipeline to send data to LaunchDarkly
52
    traces/ld:
53
      receivers: [otlp]
54
      processors:
55
        [
56
          filter/launchdarkly-spanevents,
57
          filter/launchdarkly-spans,
58
          groupbytrace,
59
          batch,
60
        ]
61
      exporters: [otlphttp/launchdarkly]

Autogenerated LaunchDarkly metrics from OpenTelemetry trace data

When LaunchDarkly receives OpenTelemetry trace data, it processes and converts this data into events that LaunchDarkly metrics track over time. Experimentation and guarded rollouts use these metrics to measure flag performance.

There are two types of events that LaunchDarkly creates from OpenTelemetry traces: route-specific events and global events. Route-specific events are useful when you are experimenting with a change that is known to impact a small subset of your server’s HTTP routes. Global events are useful when you believe your change may impact all routes, or when you are not sure of the impact of your change.

LaunchDarkly automatically creates the following metrics from the events that LaunchDarkly produces from your OpenTelemetry trace data. This trace data includes the feature flag and the context for which you evaluated the flag. You can also create these metrics manually if you wish. To learn more, read Metrics autogenerated from server-side SDKs using OpenTelemetry.

LaunchDarkly supports both the 1.20.0 and 1.23.0 versions of the OpenTelemetry semantic conventions, so you can use http.request and http.status_code instead of http.request.method and http.response.status_code if you are using 1.20.0. To learn more, read OpenTelemetry’s HTTP semantic convention migration guide.

When you create a per-route metric, you must provide the http.request.method and http.route for each metric in the string templates specified in the table above. It is important that the method and route match the corresponding HTTP semantic convention attributes in the spans you are sending to LaunchDarkly.