OpenTelemetry in server-side SDKs

If you use LaunchDarkly SDKs, resource attributes are set automatically. You do not need set the resource attributes yourself, and you can skip this section.

If you use only OpenTelemetry SDKs, you must include your LaunchDarkly SDK key as a resource attribute in your telemetry data, so that the data is correctly associated with your LaunchDarkly project and environment. You have several options for how to set the resource attribute.

To set the resource attribute using an environment variable, use the OTEL_RESOURCE_ATTRIBUTES environment variable:

$
export OTEL_RESOURCE_ATTRIBUTES="highlight.project_id=$YOUR_SDK_KEY"

To set the resource attribute in your OpenTelemetry collector configuration, use the resource processor:

1
# excerpt from otel-collector-config.yaml
2
processors:
3
  resource:
4
    attributes:
5
      - key: highlight.project_id
6
        value: YOUR_SDK_KEY
7
        action: upsert

For a complete example of a collector configuration, read Configuring the collector, below.

To set the resource attribute programmatically in your application code, use the OpenTelemetry resource available for your language. Here are a few examples:

1
import (
2
    "go.opentelemetry.io/otel/sdk/resource"
3
    "go.opentelemetry.io/otel/sdk/trace"
4
)
5
6
res := resource.NewWithAttributes(
7
    semconv.SchemaURL,
8
    attribute.String("highlight.project_id", "YOUR_SDK_KEY"),
9
)
10
11
tp := trace.NewTracerProvider(
12
    trace.WithResource(res),
13
)

To configure a pipeline to send trace data to LaunchDarkly, modify the collector’s config file, otel-collector-config.yaml, as follows:

1
# The receivers specify how the collector receives data.
2
# In this example, it receives data using the OpenTelemetry Protocol (OTLP) over gRPC and HTTP.
3
receivers:
4
  otlp:
5
    protocols:
6
      grpc:
7
      http:
8
9
# The exporters specify how the collector sends data.
10
exporters:
11
  otlphttp:
12
    endpoint: https://otel.observability.app.launchdarkly.com:4318
13
  otlp:
14
    endpoint: https://otel.observability.app.launchdarkly.com:4317
15
16
# The processors specify how the collector processes the trace data.
17
# In this example, it groups spans that belong to the same trace,
18
# and only passes along span events related to feature flags or exceptions.
19
processors:
20
21
  # This processor sets the resource attribute to your SDK key.
22
  # If you are using a LaunchDarkly SDK, you do NOT need to include this.
23
  # If you are using an OpenTelemetry SDK, you do need to set the
24
  # resource attributes. You can set them here, set them in your application,
25
  # or set them using an environment variable.
26
  resource:
27
    attributes:
28
        - key: highlight.project_id
29
          value: YOUR_SDK_KEY
30
          action: upsert
31
32
  # The groupbytrace processor groups all spans that belong to the same trace together.
33
  # This is required to ensure LaunchDarkly receives a complete trace in a single request.
34
  groupbytrace:
35
    wait_duration: 10s
36
37
  # This filter removes all span events that are not "feature_flag" or "exception".
38
  # It is optional, but useful to limit the amount of data you send to LaunchDarkly.
39
  filter/launchdarkly-spanevents:
40
    error_mode: ignore
41
    traces:
42
      spanevent:
43
        - 'not (name == "feature_flag" or name == "exception")'
44
45
  # This filter removes all spans that do not have an HTTP route or any span events
46
  # remaining after the previous filter has been applied
47
  filter/launchdarkly-spans:
48
    error_mode: ignore
49
    traces:
50
      span:
51
        - 'not (attributes["http.route"] != nil or Len(events) > 0)'
52
53
  batch:
54
55
extensions:
56
  health_check:
57
58
service:
59
  pipelines:
60
    # Add a new pipeline to send data to LaunchDarkly
61
    traces/ld:
62
      receivers: [otlp]
63
      processors:
64
        [
65
          filter/launchdarkly-spanevents,
66
          filter/launchdarkly-spans,
67
          groupbytrace,
68
          batch,
69
        ]
70
      exporters: [otlphttp/launchdarkly]
71
    metrics:
72
      receivers: [otlp]
73
      processors: [resource]
74
      exporters: [otlphttp]
75
    logs:
76
      receivers: [otlp]
77
      processors: [resource]
78
      exporters: [otlphttp]

To add flag evaluation information to OpenTelemetry spans, first add the Telemetry package from NuGet:

$
dotnet add package LaunchDarkly.ServerSdk.Telemetry

Next, import the LaunchDarkly.Sdk.Server.Hooks and LaunchDarkly.Sdk.Server.Telemetry namespaces. Then you can create a new tracing hook and reference it in the configuration options when you initialize the SDK client.

Here’s how:

1
using LaunchDarkly.Sdk.Server.Hooks;
2
using LaunchDarkly.Sdk.Server.Telemetry;
3
4
var config = Configuration.Builder("sdk-key-123abc")
5
  .Hooks(Components.Hooks()
6
    .Add(TracingHook.Default())
7
  ).Build();
8
9
var client = new LdClient(config);

When this configuration is set, the SDK adds span events for each call to a *Variation* method to your active span. The span event information contains details of each flag evaluation, including the flag key and the context key. You can collect this information in Guarded rollouts or other observability tools.

Optionally, you can configure the tracing hook to:

• create new spans that contain span events with flag evaluation information, rather than adding the events to the active span. The new spans are nested in the current active span, and are not active.
• include the flag’s evaluated flag value in the span event.

Here’s how:

1
using LaunchDarkly.Sdk.Server.Hooks;
2
using LaunchDarkly.Sdk.Server.OpenTelemetry;
3
4
var config = Configuration.Builder("sdk-key-123abc")
5
  .Hooks(Components.Hooks()
6
    .Add(TracingHook.Builder()
7
      .CreateActivities()
8
      .IncludeVariant()
9
      .Build()
10
    )
11
  ).Build();
12
13
var client = new LdClient(config);

Starting with v1.1 of the LaunchDarkly.Sdk.Server.Hooks package, which requires v8.7+ of the .NET (server-side) SDK, LaunchDarkly automatically decorates any OpenTelemetry spans with project and environment information.

When you create the hook, you can optionally pass in a client-side ID to specify the environment. Use the builder’s .EnvironmentId() method and pass in the client-side ID. To learn more about client-side IDs, read Keys.

Because LaunchDarkly automatically decorates the OpenTelemetry spans, you should not need to call this method explicitly. The exception is if you are using persistent stores, in which case you must either call .EnvironmentId() or use the single-environment OpenTelemetry Collector configuration to ensure that your OpenTelemetry spans have correct project and environment information.

Here’s how:

1
using LaunchDarkly.Sdk.Server.Hooks; // v1.1 or later
2
using LaunchDarkly.Sdk.Server.OpenTelemetry;
3
4
var config = Configuration.Builder("sdk-key-123abc")
5
  .Hooks(Components.Hooks()
6
    .Add(TracingHook.Builder()
7
      .EnvironmentId("client-side-id-123abc")
8
    )
9
  ).Build();
10
11
var client = new LdClient(config);

In the .NET (server-side) SDK, the tracing hook is implemented using Microsoft’s System.Diagnostics APIs. Therefore, some of the terminology it uses differs from the OpenTelemetry standard. For example, the tracing hook in the .NET (server-side) SDK uses the method CreateActivities(), rather than CreateSpans(), to enable child spans.

To learn more, read TracingHookBuilder and ConfigurationBuilder.

To add flag evaluation information to OpenTelemetry spans, import the ldhooks package. Then, create a new tracing hook and reference it in the configuration options when you initialize the SDK client.

Here’s how:

1
import (
2
  ld "github.com/launchdarkly/go-server-sdk/v7"
3
  "github.com/launchdarkly/go-server-sdk/v7/ldhooks"
4
  "github.com/launchdarkly/go-server-sdk/ldotel"
5
)
6
7
var config ld.Config
8
9
client, _ = ld.MakeCustomClient("sdk-key-123abc",
10
  ld.Config{
11
    Hooks: []ldhooks.Hook{ldotel.NewTracingHook()},
12
  }, 5*time.Second)

When this configuration is set, the SDK adds span events for each call to a *Variation*Ctx method to your active span.

The span event information contains details of each flag evaluation, including the flag key and the context key. You can collect this information in Guarded rollouts or other observability tools.

The *Variation*Ctx methods are the same as the *Variation* methods, except that they also require a Go context. For example, BoolVariationCtx and BoolVariationDetailCtx are the same as the BoolVariation and BoolVariationDetail methods, except that they also require a Go context. This Go context is used in the hook implementation. If you are using hooks, you must update your variation calls to use the Ctx methods.

Optionally, you can configure the tracing hook to:

• create new spans that contain span events with flag evaluation information, rather than adding the events to the active span. The new spans are nested in the current active span, and are not active.
• include the flag’s evaluated flag value in the span event.

Here’s how:

1
import (
2
  ld "github.com/launchdarkly/go-server-sdk/v7"
3
  "github.com/launchdarkly/go-server-sdk/v7/ldhooks"
4
  "github.com/launchdarkly/go-server-sdk/ldotel"
5
)
6
7
var config ld.Config
8
9
client, _ = ld.MakeCustomClient("sdk-key-123abc",
10
  ld.Config{
11
    Hooks: []ldhooks.Hook{ldotel.NewTracingHook(ldotel.WithSpans(), ldotel.WithVariant())},
12
  }, 5*time.Second)

To learn more, read Config.

To add flag evaluation information to OpenTelemetry spans, import the LaunchDarkly Java Otel Hook package. Then, create a new tracing hook and reference it in the configuration options when you initialize the SDK client.

Here’s how:

1
import com.launchdarkly.sdk.*;
2
import com.launchdarkly.sdk.server.*;
3
import com.launchdarkly.integrations.*;
4
5
TracingHook tracingHook = new TracingHook.Builder().build();
6
7
LDConfig config = new LDConfig.Builder()
8
    .hooks(
9
        Components.hooks().setHooks(Collections.singletonList(tracingHook)))
10
    .build();
11
12
LDClient client = new LDClient("sdk-key-123abc", config);

When this configuration is set, the SDK adds span events for each call to a *Variation* method to your active span. The span event information contains details of each flag evaluation, including the flag key and the context key. You can collect this information in Guarded rollouts or other observability tools.

Optionally, you can configure the tracing hook to:

• create new spans that contain span events with flag evaluation information, rather than adding the events to the active span. The new spans are nested in the current active span, and are not active.
• include the flag’s evaluated flag value in the span event.

Here’s how:

1
import com.launchdarkly.sdk.*;
2
import com.launchdarkly.sdk.server.*;
3
import com.launchdarkly.integrations.*;
4
5
TracingHook tracingHook = new TracingHook.Builder().withSpans().withVariant().build();
6
7
LDConfig config = new LDConfig.Builder()
8
    .hooks(
9
        Components.hooks().setHooks(Collections.singletonList(tracingHook)))
10
    .build();
11
12
LDClient client = new LDClient("sdk-key-123abc", config);

To learn more, read LDConfig.Builder.

To add flag evaluation information to OpenTelemetry spans, import the TracingHook package. Then, create a new tracing hook and reference it in the configuration options when you initialize the SDK client.

Here’s how:

1
import { LDOptions, init } from '@launchdarkly/node-server-sdk';
2
import { TracingHook } from '@launchdarkly/node-server-sdk-otel';
3
4
const options: LDOptions = {
5
  hooks: [new TracingHook()]
6
}
7
8
const client = init('sdk-key-123abc', options);

When this configuration is set, the SDK adds span events for each call to a *Variation* method to your active span. The span event information contains details of each flag evaluation, including the flag key and the context key. You can collect this information in Guarded rollouts or other observability tools.

Optionally, you can configure the tracing hook to:

• create new spans that contain span events with flag evaluation information, rather than adding the events to the active span. The new spans are nested in the current active span, and are not active.
• include the flag’s evaluated flag value in the span event.

Here’s how:

1
import { LDOptions, init } from '@launchdarkly/node-server-sdk';
2
import { TracingHook } from '@launchdarkly/node-server-sdk-otel';
3
4
const options: LDOptions = {
5
  hooks: [new TracingHook({spans: true, includeVariant: true})]
6
}
7
8
const client = init('sdk-key-123abc', options);

Starting with v1.2 of the @LaunchDarkly/node-server-sdk-otel package, which requires v9.9+ of the Node.js (server-side) SDK, LaunchDarkly automatically decorates any OpenTelemetry spans with project and environment information.

When you create the hook, you can optionally pass in a client-side ID to specify the environment. Use the environmentId option when you create the new TracingHook. To learn more about client-side IDs, read Keys.

Because LaunchDarkly automatically decorates the OpenTelemetry spans, you should not need to set this option explicitly. The exception is if you are using persistent stores, in which case you must either set the environmentId or use the single-environment OpenTelemetry Collector configuration to ensure that your OpenTelemetry spans have correct project and environment information.

Here’s how:

1
import { LDOptions, init } from '@launchdarkly/node-server-sdk';
2
import { TracingHook } from '@launchdarkly/node-server-sdk-otel'; // v1.2 or later
3
4
const options: LDOptions = {
5
  hooks: [new TracingHook({
6
    spans: true,
7
    includeVariant: true,
8
    environmentId: 'client-side-id-123abc'
9
  })]
10
}
11
12
const client = init('sdk-key-123abc', options);

To learn more, read LDOptions.

To add flag evaluation information to OpenTelemetry spans, first install the package:

$
pip install launchdarkly-server-sdk-otel

Next, import Hook and HookOptions from the package. Then you can create a new tracing hook and reference it in the configuration options when you initialize the SDK client.

Here’s how:

1
import ldclient
2
from ldclient import Config
3
from ldotel.tracing import Hook, HookOptions
4
5
config = Config(sdk_key, hooks=[Hook()])
6
ldclient.set_config(config=config)
7
client = ldclient.get()

When this configuration is set, the SDK adds span events for each call to a *variation* method to your active span. The span event information contains details of each flag evaluation, including the flag key and the context key. You can collect this information in Guarded rollouts or other observability tools.

Optionally, you can configure the tracing hook to:

• create new spans. The new spans are nested in the current active span, and are not active. The span events with flag evaluation information are still added to the parent span.
• include the flag’s evaluated flag value in the span event.

Here’s how:

1
import ldclient
2
from ldclient.config import Config
3
4
from ldotel.tracing import Hook, HookOptions
5
6
config = Config(sdk_key, hooks=[Hook(HookOptions(add_spans=True, include_variant=True))])
7
ldclient.set_config(config=config)
8
client = ldclient.get()

To learn more, read Config.

To add flag evaluation information to OpenTelemetry spans, first install the gem:

$
gem install launchdarkly-server-sdk-otel

Then, you can create a new tracing hook and reference it in the configuration options when you initialize the SDK client.

Here’s how:

1
require 'ldclient-otel'
2
3
config = LaunchDarkly::Config.new(hooks: [LaunchDarkly::Otel::TracingHook.new])
4
5
client = LaunchDarkly::LDClient.new("sdk-key-123abc", config);

When this configuration is set, the SDK adds span events for each call to a *Variation* method to your active span. The span event information contains details of each flag evaluation, including the flag key and the context key. You can collect this information in Guarded rollouts or other observability tools.

Optionally, you can configure the tracing hook to:

• create new spans. The new spans are nested in the current active span, and are not active. The span events with flag evaluation information are still added to the parent span.
• include the flag’s evaluated flag value in the span event.

Here’s how:

1
require 'ldclient-otel'
2
3
tracing_hook_options = LaunchDarkly::Otel::TracingHookOptions.new(add_spans: true, include_variant: true)
4
hook = LaunchDarkly::Otel::TracingHook.new(tracing_hook_options)
5
config = LaunchDarkly::Config.new(hooks: [hook])
6
7
client = LaunchDarkly::LDClient.new("sdk-key-123abc", config);

To learn more, read Config.