Sending OpenTelemetry traces to LaunchDarkly

This feature is for Early Access Program customers only

Sending OpenTelemetry traces to LaunchDarkly is only available to members of LaunchDarkly’s Early Access Program (EAP). If you want access to this feature, join the EAP.

Enabling OpenTelemetry traces in your server-side SDKs is available to all customers. To learn more, read OpenTelemetry.

Overview

This topic explains how to send OpenTelemetry traces to LaunchDarkly. LaunchDarkly converts this data into events that LaunchDarkly metrics track over time. Experimentation and guarded rollouts use these metrics to measure flag performance.

This topic covers:

About OpenTelemetry

OpenTelemetry (OTel) is an open source observability framework and toolkit designed to create and manage telemetry data such as traces, metrics, and logs. You can use LaunchDarkly’s OpenTelemetry protocol (OTLP) endpoint to send OpenTelemetry traces to LaunchDarkly.

LaunchDarkly converts the information contained in these traces to produce events for use with Experimentation and guarded rollouts. Typically the information you collect in the traces is related to API performance and errors.

Because OpenTelemetry is vendor- and tool-agnostic, you can reuse the instrumentation that may already exist in your code to create LaunchDarkly metrics without changing application code.

LaunchDarkly supports receiving trace data over gRPC, HTTP/protobuf, and HTTP/JSON.

Send trace data to LaunchDarkly

To send OpenTelemetry trace data to LaunchDarkly, we recommend using the OpenTelemetry Collector. The OpenTelemetry Collector is a vendor-agnostic proxy that can receive, process, and export telemetry data. You can configure an OpenTelemetry pipeline with a processor to filter out data that is not relevant to LaunchDarkly, reducing data transport costs, and to send data to LaunchDarkly in addition to other OTLP backends that you may already be using.

LaunchDarkly processes two types of data from your OpenTelemetry traces:

  • HTTP span attributes, including latency, 5xx occurrences, and other errors, where the span has or overlaps with another span that has at least one feature flag span event
  • exception span events that occur in the wake of a feature flag span event

A feature flag span event is defined as any span event that contains a feature_flag.context.key attribute. LaunchDarkly ignores traces that do not include span events with this attribute.

To ensure your spans have compatible feature flag events, configure your SDK to use the OpenTelemetry tracing hook. This hook automatically attaches feature flag event data to your OTel traces for you.

Try it in your SDK: OpenTelemetry

Generate a new LaunchDarkly service token

All trace data sent to LaunchDarkly must be authenticated with a LaunchDarkly personal or service API access token with a Writer role, or a custom role that allows the token to perform the importEventData action on the LaunchDarkly environment you are targeting.

Here’s how to create an API service token with the appropriate inline policy:

  1. Click the gear icon in the left sidenav to view Organization settings. The General settings page appears.

  2. Click Authorization.

  3. Click Create token. The “Create access token” dialog appears.

  4. Give your token a human-readable Name, such as “Compass integration.”

  5. Give your token an “Inline policy” Role.

  6. Click the Advanced editor button and specify the following policy in the Editor field to allow the token to perform the importEventData action on all LaunchDarkly environment resources:

    OpenTelemetry Import Inline policy
    1[
    2  {
    3    "resources": [
    4      "proj/*:env/*"
    5    ],
    6    "actions": [
    7      "importEventData"
    8    ],
    9    "effect": "allow"
    10  }
    11]

  7. Select This is a service token.

The inline policy for the OpenTelemetry import access token.

The inline policy for the OpenTelemetry import access token.


  1. Click Save token and copy the newly created token to your clipboard. You will need it to configure the OpenTelemetry collector in the following section.

Collector configuration

Collector pipelines consists of three components: receivers, exporters, and processors. Because the exporter is sending data to LaunchDarkly, it requires a LaunchDarkly access token. To learn how to create an access token, read API access tokens.

To configure a pipeline to send trace data to LaunchDarkly, modify the Collector’s config file, otel-collector-config.yaml, as follows:

1# The receivers specify how the Collector receives data.
2# In this example, it receives data using the OpenTelemetry Protocol (OTLP) over gRPC and HTTP.
3receivers:
4  otlp:
5    protocols:
6      grpc:
7      http:
8
9# The exporters specify how the Collector sends data.
10# In this example, it sends data to LaunchDarkly using HTTP.
11exporters:
12  otlphttp/launchdarkly:
13    traces_endpoint: https://events.launchdarkly.com/otlp/traces
14    headers:
15      # The Authorization header must be a personal or service API access token with the `importEventData` action.
16      Authorization: ${env:LD_ACCESS_TOKEN}
17      X-LaunchDarkly-Project: example-project-key
18      X-LaunchDarkly-Environment: example-environment-key
19
20# The processors specify how the Collector processes the trace data.
21# In this example, it groups spans that belong to the same trace,
22# and only passes along span events related to feature flags or exceptions.
23processors:
24
25  # The groupbytrace processor groups all spans that belong to the same trace together.
26  # This is required to ensure LaunchDarkly receives a complete trace in a single request.
27  groupbytrace:
28    wait_duration: 10s
29
30  # Remove all span events that are not "feature_flag" or "exception"
31  filter/launchdarkly-spanevents:
32    error_mode: ignore
33    traces:
34      spanevent:
35        - 'not (name == "feature_flag" or name == "exception")'
36
37  # Remove all spans that do not have an HTTP route or any span events remaining
38  # after the previous filter has been applied
39  filter/launchdarkly-spans:
40    error_mode: ignore
41    traces:
42      span:
43        - 'not (attributes["http.route"] != nil or Len(events) > 0)'
44
45  batch:
46
47extensions:
48  health_check:
49
50service:
51  pipelines:
52    # Add a new pipeline to send data to LaunchDarkly
53    traces/ld:
54      receivers: [otlp]
55      processors:
56        [
57          filter/launchdarkly-spanevents,
58          filter/launchdarkly-spans,
59          groupbytrace,
60          batch,
61        ]
62      exporters: [otlphttp/launchdarkly]

Autogenerated LaunchDarkly metrics from OpenTelemetry trace data

When LaunchDarkly receives OpenTelemetry trace data, it processes and converts this data into events that LaunchDarkly metrics track over time. Experimentation and guarded rollouts use these metrics to measure flag performance.

There are two types of events that LaunchDarkly creates from OpenTelemetry traces: route-specific events and global events. Route-specific events are useful when you are experimenting with a change that is known to impact a small subset of your server’s HTTP routes. Global events are useful when you believe your change may impact all routes, or when you are not sure of the impact of your change.

LaunchDarkly automatically creates the following metrics from the events that LaunchDarkly produces from your OpenTelemetry trace data. You can also create these metrics manually if you wish. To learn more, read Metrics autogenerated from server-side SDKs using OpenTelemetry.

Event typeLaunchDarkly metric propertiesEvent name template and examples
per-route HTTP request latency
  • Event kind: custom numeric
  • Unit of measure: ms
  • Success criteria: lower
http.latency;method={http.request.method};route={http.route}

Examples:
http.latency;method=GET;route=/api/v2/flags
http.latency;method=PATCH;route=/api/v2/flags/{id}
global HTTP request latency
  • Event kind: custom numeric
  • Unit of measure: ms
  • Success criteria: lower
otel.http.latency

Example:
otel.http.latency
per-route HTTP request errors
  • Event kind: custom conversion
  • Success criteria: lower
http.error;method={http.request.method};route={http.route}

Examples:
http.error;method=GET;route=/api/v2/flags
http.error;method=PATCH;route=/api/v2/flags/{id}
global exceptions
  • Event kind: custom conversion
  • Success criteria: lower
otel.exception

Example:
otel.exception
global HTTP 5xxs
  • Event kind: custom conversion
  • Success criteria: lower
otel.http.5XX

Example:
otel.http.5XX
per-route HTTP 5xxs
  • Event kind: custom conversion
  • Success criteria: lower
http.5XX;method={http.request.method};route={http.route}

Examples:
http.5XX;method=GET;route=/api/v2/flags
http.5XX;method=PATCH;route=/api/v2/flags/{id}

LaunchDarkly supports both the 1.20.0 and 1.23.0 versions of the OpenTelemetry semantic conventions, so you can use http.request and http.status_code instead of http.request.method and http.response.status_code if you are using 1.20.0. To learn more, read OpenTelemetry’s HTTP semantic convention migration guide.

When you create a per-route metric, you must provide the http.request.method and http.route for each metric in the string templates specified in the table above. It is important that the method and route match the corresponding HTTP semantic convention attributes in the spans you are sending to LaunchDarkly.

Built with