OpenTelemetry in server-side SDKs

If you use LaunchDarkly SDKs, resource attributes are set automatically. You do not need set the resource attributes yourself, and you can skip this section.

If you use only OpenTelemetry SDKs, you must include an active LaunchDarkly SDK key as a resource attribute in your telemetry data, so that the data is correctly associated with your LaunchDarkly project and environment. You have several options for how to set the resource attribute.

To set the resource attribute using an environment variable, use the OTEL_RESOURCE_ATTRIBUTES environment variable:

$
export OTEL_RESOURCE_ATTRIBUTES="launchdarkly.project_id=$YOUR_SDK_KEY"

To set the resource attribute in your OpenTelemetry collector configuration, use the resource processor:

1
# excerpt from otel-collector-config.yaml
2
processors:
3
  resource:
4
    attributes:
5
      - key: launchdarkly.project_id
6
        value: YOUR_SDK_KEY
7
        action: upsert

For a complete example of a collector configuration, read Configuring the collector, below.

To set the resource attribute programmatically in your application code, use the OpenTelemetry resource available for your language. Here are a few examples:

1
import (
2
    "go.opentelemetry.io/otel/sdk/resource"
3
    "go.opentelemetry.io/otel/sdk/trace"
4
)
5
6
res := resource.NewWithAttributes(
7
    semconv.SchemaURL,
8
    attribute.String("launchdarkly.project_id", "YOUR_SDK_KEY"),
9
)
10
11
tp := trace.NewTracerProvider(
12
    trace.WithResource(res),
13
)

To configure a pipeline to send trace data to LaunchDarkly, modify the collector’s config file, otel-collector-config.yaml, as follows:

1
# The receivers specify how the collector receives data.
2
# In this example, it receives data using the OpenTelemetry Protocol (OTLP) over gRPC and HTTP.
3
receivers:
4
  otlp:
5
    protocols:
6
      grpc:
7
      http:
8
9
# The exporters specify how the collector sends data.
10
exporters:
11
  otlphttp:
12
    endpoint: https://otel.observability.app.launchdarkly.com:4318
13
  otlp:
14
    endpoint: https://otel.observability.app.launchdarkly.com:4317
15
16
# The processors specify how the collector processes the trace data.
17
# In this example, it drops spans that aren't needed for guarded rollouts.
18
processors:
19
20
  # This processor sets the resource attribute to your SDK key.
21
  # If you are using a LaunchDarkly SDK, you do NOT need to include this.
22
  # If you are using an OpenTelemetry SDK, you do need to set the
23
  # resource attributes. You can set them here, set them in your application,
24
  # or set them using an environment variable.
25
  resource:
26
    attributes:
27
        - key: launchdarkly.project_id
28
          value: YOUR_SDK_KEY
29
          action: upsert
30
31
  # This filter drops all span events except for exceptions and feature flag evaluations
32
  # that were part of an active guarded rollout.
33
  # It is optional, but useful to limit the amount of data you send to LaunchDarkly.
34
  filter/launchdarkly-spanevents:
35
    error_mode: ignore
36
    traces:
37
      spanevent:
38
        - 'not ((name == "feature_flag" and attributes["feature_flag.result.reason.inExperiment"] == true) or name == "exception")'
39
40
  # This filter removes all spans that do not have an HTTP route or any span events
41
  # remaining after the previous filter has been applied
42
  filter/launchdarkly-spans:
43
    error_mode: ignore
44
    traces:
45
      span:
46
        - 'not (attributes["http.route"] != nil or Len(events) > 0)'
47
48
  batch:
49
50
extensions:
51
  health_check:
52
53
service:
54
  pipelines:
55
    # Add a new pipeline to send data to LaunchDarkly
56
    traces/ld:
57
      receivers: [otlp]
58
      processors:
59
        [
60
          filter/launchdarkly-spanevents,
61
          filter/launchdarkly-spans,
62
          batch,
63
        ]
64
      exporters: [otlphttp/launchdarkly]
65
    metrics:
66
      receivers: [otlp]
67
      processors: [resource]
68
      exporters: [otlphttp]
69
    logs:
70
      receivers: [otlp]
71
      processors: [resource]
72
      exporters: [otlphttp]

To add flag evaluation information to OpenTelemetry spans, first add the Telemetry package from NuGet:

$
dotnet add package LaunchDarkly.ServerSdk.Telemetry

Next, import the LaunchDarkly.Sdk.Server.Hooks and LaunchDarkly.Sdk.Server.Telemetry namespaces. Then you can create a new tracing hook and reference it in the configuration options when you initialize the SDK client.

Here’s how:

1
using LaunchDarkly.Sdk.Server.Hooks;
2
using LaunchDarkly.Sdk.Server.Telemetry;
3
4
var config = Configuration.Builder("YOUR_SDK_KEY")
5
  .Hooks(Components.Hooks()
6
    .Add(TracingHook.Default())
7
  ).Build();
8
9
var client = new LdClient(config);

When this configuration is set, the SDK adds span events for each call to a *Variation* method to your active span. The span event information contains details of each flag evaluation, including the flag key and the context key. You can collect this information in Guarded rollouts or other observability tools.

Optionally, you can configure the tracing hook to:

• create new spans that contain span events with flag evaluation information, rather than adding the events to the active span. The new spans are nested in the current active span, and are not active.
• include the flag’s evaluated flag value in the span event.

Here’s how:

1
using LaunchDarkly.Sdk.Server.Hooks;
2
using LaunchDarkly.Sdk.Server.OpenTelemetry;
3
4
var config = Configuration.Builder("YOUR_SDK_KEY")
5
  .Hooks(Components.Hooks()
6
    .Add(TracingHook.Builder()
7
      .CreateActivities()
8
      .IncludeValue()
9
      .Build()
10
    )
11
  ).Build();
12
13
var client = new LdClient(config);

Starting with v1.1 of the LaunchDarkly.Sdk.Server.Hooks package, which requires v8.7+ of the .NET (server-side) SDK, LaunchDarkly automatically decorates any OpenTelemetry spans with project and environment information.

When you create the hook, you can optionally pass in a client-side ID to specify the environment. Use the builder’s .EnvironmentId() method and pass in the client-side ID. To learn more about client-side IDs, read Keys.

Because LaunchDarkly automatically decorates the OpenTelemetry spans, you should not need to call this method explicitly. The exception is if you are using persistent stores, in which case you must either call .EnvironmentId() or use the single-environment OpenTelemetry Collector configuration to ensure that your OpenTelemetry spans have correct project and environment information.

Here’s how:

1
using LaunchDarkly.Sdk.Server.Hooks; // v1.1 or later
2
using LaunchDarkly.Sdk.Server.OpenTelemetry;
3
4
var config = Configuration.Builder("YOUR_SDK_KEY")
5
  .Hooks(Components.Hooks()
6
    .Add(TracingHook.Builder()
7
      .EnvironmentId("example-client-side-id")
8
    )
9
  ).Build();
10
11
var client = new LdClient(config);

In the .NET (server-side) SDK, the tracing hook is implemented using Microsoft’s System.Diagnostics APIs. Therefore, some of the terminology it uses differs from the OpenTelemetry standard. For example, the tracing hook in the .NET (server-side) SDK uses the method CreateActivities(), rather than CreateSpans(), to enable child spans.

To learn more, read TracingHookBuilder and ConfigurationBuilder.

To add flag evaluation information to OpenTelemetry spans, import the ldhooks package. Then, create a new tracing hook and reference it in the configuration options when you initialize the SDK client.

Here’s how:

1
import (
2
  ld "github.com/launchdarkly/go-server-sdk/v7"
3
  "github.com/launchdarkly/go-server-sdk/v7/ldhooks"
4
  "github.com/launchdarkly/go-server-sdk/ldotel"
5
)
6
7
var config ld.Config
8
9
client, _ = ld.MakeCustomClient("YOUR_SDK_KEY",
10
  ld.Config{
11
    Hooks: []ldhooks.Hook{ldotel.NewTracingHook()},
12
  }, 5*time.Second)

When this configuration is set, the SDK adds span events for each call to a *Variation*Ctx method to your active span.

The span event information contains details of each flag evaluation, including the flag key and the context key. You can collect this information in Guarded rollouts or other observability tools.

The *Variation*Ctx methods are the same as the *Variation* methods, except that they also require a Go context. For example, BoolVariationCtx and BoolVariationDetailCtx are the same as the BoolVariation and BoolVariationDetail methods, except that they also require a Go context. This Go context is used in the hook implementation. If you are using hooks, you must update your variation calls to use the Ctx methods. If you are working with the LDScopedClient, it also includes *Variation*Ctx methods. These require a Go context and, like other LDScopedClient methods, do not require a LaunchDarkly context.

Optionally, you can configure the tracing hook to:

• create new spans that contain span events with flag evaluation information, rather than adding the events to the active span. The new spans are nested in the current active span, and are not active.
• include the flag’s evaluated flag value in the span event.

Here’s how:

1
import (
2
  ld "github.com/launchdarkly/go-server-sdk/v7"
3
  "github.com/launchdarkly/go-server-sdk/v7/ldhooks"
4
  "github.com/launchdarkly/go-server-sdk/ldotel"
5
)
6
7
var config ld.Config
8
9
client, _ = ld.MakeCustomClient("YOUR_SDK_KEY",
10
  ld.Config{
11
    Hooks: []ldhooks.Hook{ldotel.NewTracingHook(ldotel.WithSpans(), ldotel.WithValue())},
12
  }, 5*time.Second)

To learn more, read Config.

To add flag evaluation information to OpenTelemetry spans, import the LaunchDarkly Java Otel Hook package. Then, create a new tracing hook and reference it in the configuration options when you initialize the SDK client.

Here’s how:

1
import com.launchdarkly.sdk.*;
2
import com.launchdarkly.sdk.server.*;
3
import com.launchdarkly.integrations.*;
4
5
TracingHook tracingHook = new TracingHook.Builder().build();
6
7
LDConfig config = new LDConfig.Builder()
8
    .hooks(
9
        Components.hooks().setHooks(Collections.singletonList(tracingHook)))
10
    .build();
11
12
LDClient client = new LDClient("YOUR_SDK_KEY", config);

When this configuration is set, the SDK adds span events for each call to a *Variation* method to your active span. The span event information contains details of each flag evaluation, including the flag key and the context key. You can collect this information in Guarded rollouts or other observability tools.

Optionally, you can configure the tracing hook to:

• create new spans that contain span events with flag evaluation information, rather than adding the events to the active span. The new spans are nested in the current active span, and are not active.
• include the flag’s evaluated flag value in the span event.

Here’s how:

1
import com.launchdarkly.sdk.*;
2
import com.launchdarkly.sdk.server.*;
3
import com.launchdarkly.integrations.*;
4
5
TracingHook tracingHook = new TracingHook.Builder().withSpans().withValue().build();
6
7
LDConfig config = new LDConfig.Builder()
8
    .hooks(
9
        Components.hooks().setHooks(Collections.singletonList(tracingHook)))
10
    .build();
11
12
LDClient client = new LDClient("YOUR_SDK_KEY", config);

To learn more, read LDConfig.Builder.

To add flag evaluation information to OpenTelemetry spans, import the TracingHook package. Then, create a new tracing hook and reference it in the configuration options when you initialize the SDK client.

Here’s how:

1
import { LDOptions, init } from '@launchdarkly/node-server-sdk';
2
import { TracingHook } from '@launchdarkly/node-server-sdk-otel';
3
4
const options: LDOptions = {
5
  hooks: [new TracingHook()]
6
}
7
8
const client = init('YOUR_SDK_KEY', options);

When this configuration is set, the SDK adds span events for each call to a *Variation* method to your active span. The span event information contains details of each flag evaluation, including the flag key and the context key. You can collect this information in Guarded rollouts or other observability tools.

Optionally, you can configure the tracing hook to:

• create new spans that contain span events with flag evaluation information, rather than adding the events to the active span. The new spans are nested in the current active span, and are not active.
• include the flag’s evaluated flag value in the span event.

Here’s how:

1
import { LDOptions, init } from '@launchdarkly/node-server-sdk';
2
import { TracingHook } from '@launchdarkly/node-server-sdk-otel';
3
4
const options: LDOptions = {
5
  hooks: [new TracingHook({spans: true, includeValue: true})]
6
}
7
8
const client = init('YOUR_SDK_KEY', options);

Starting with v1.2 of the @LaunchDarkly/node-server-sdk-otel package, which requires v9.9+ of the Node.js (server-side) SDK, LaunchDarkly automatically decorates any OpenTelemetry spans with project and environment information.

When you create the hook, you can optionally pass in a client-side ID to specify the environment. Use the environmentId option when you create the new TracingHook. To learn more about client-side IDs, read Keys.

Because LaunchDarkly automatically decorates the OpenTelemetry spans, you should not need to set this option explicitly. The exception is if you are using persistent stores, in which case you must either set the environmentId or use the single-environment OpenTelemetry Collector configuration to ensure that your OpenTelemetry spans have correct project and environment information.

Here’s how:

1
import { LDOptions, init } from '@launchdarkly/node-server-sdk';
2
import { TracingHook } from '@launchdarkly/node-server-sdk-otel'; // v1.2 or later
3
4
const options: LDOptions = {
5
  hooks: [new TracingHook({
6
    spans: true,
7
    includeValue: true,
8
    environmentId: 'example-client-side-id'
9
  })]
10
}
11
12
const client = init('YOUR_SDK_KEY', options);

To learn more, read LDOptions.

To add flag evaluation information to OpenTelemetry spans, first install the package:

$
composer require launchdarkly/server-sdk-otel

The package depends on the OpenTelemetry API. You also need a runtime OpenTelemetry SDK implementation, such as open-telemetry/sdk, and at least one exporter (for example, open-telemetry/exporter-otlp for OTLP/HTTP).

Next, import the TracingHook class. Then create a new tracing hook and reference it in the configuration options when you initialize the SDK client.

Here’s how:

1
require 'vendor/autoload.php';
2
3
use LaunchDarkly\LDClient;
4
use LaunchDarkly\OpenTelemetry\TracingHook;
5
6
$client = new LDClient('YOUR_SDK_KEY', [
7
    'hooks' => [new TracingHook()],
8
]);

When this configuration is set, the SDK adds a feature_flag span event to your active OpenTelemetry span for each call to variation or variationDetail. The span event information contains details of each flag evaluation, including the flag key and the context key. You can collect this information in Guarded rollouts or other observability tools.

If no OpenTelemetry span is active when the variation call runs, the hook is a no-op.

Optionally, you can configure the tracing hook to:

• include the evaluated flag value in the span event.
• create new spans that wrap each variation call, in addition to the span event. The new spans are nested in the current active span, and are not active. This feature is experimental, and the span structure or naming may change in future releases.

Here’s how:

1
require 'vendor/autoload.php';
2
3
use LaunchDarkly\LDClient;
4
use LaunchDarkly\OpenTelemetry\TracingHook;
5
use LaunchDarkly\OpenTelemetry\TracingHookOptions;
6
7
$client = new LDClient('YOUR_SDK_KEY', [
8
    'hooks' => [
9
        new TracingHook(new TracingHookOptions(
10
            includeValue: true,
11
            addSpans: true,
12
        )),
13
    ],
14
]);

When the LaunchDarkly PHP SDK fetches flags directly from LaunchDarkly, it captures the environment ID from the polling response and the tracing hook automatically adds it to the feature_flag.set.id attribute on each span event. You should not need to set this explicitly. The exception is if you are using daemon mode or another persistent store configuration, where flags are read from a shared store rather than fetched from LaunchDarkly directly. In those cases, the polling-response metadata is not available, so you must either set the environmentId option on TracingHookOptions or use the single-environment OpenTelemetry Collector configuration to ensure that your OpenTelemetry spans have correct project and environment information.

When you create the hook, you can optionally pass in a client-side ID to specify the environment. Use the environmentId option on TracingHookOptions and pass in the client-side ID. The configured value takes precedence over the value the SDK derives from the polling response. To learn more about client-side IDs, read Keys.

Here’s how:

1
$client = new LDClient('YOUR_SDK_KEY', [
2
    'hooks' => [
3
        new TracingHook(new TracingHookOptions(
4
            environmentId: 'example-client-side-id',
5
        )),
6
    ],
7
]);

Under PHP-FPM and similar request-response SAPIs, register a shutdown function to flush your tracer provider before the request ends, or use a SimpleSpanProcessor to export spans synchronously. To learn more, read the launchdarkly/server-sdk-otel README.

To learn more, read the SDK API documentation.

To add flag evaluation information to OpenTelemetry spans, first install the package:

$
pip install launchdarkly-server-sdk-otel

Next, import Hook and HookOptions from the package. Then you can create a new tracing hook and reference it in the configuration options when you initialize the SDK client.

Here’s how:

1
import ldclient
2
from ldclient import Config
3
from ldotel.tracing import Hook, HookOptions
4
5
config = Config(sdk_key, hooks=[Hook()])
6
ldclient.set_config(config=config)
7
client = ldclient.get()

When this configuration is set, the SDK adds span events for each call to a *variation* method to your active span. The span event information contains details of each flag evaluation, including the flag key and the context key. You can collect this information in Guarded rollouts or other observability tools.

Optionally, you can configure the tracing hook to:

• create new spans. The new spans are nested in the current active span, and are not active. The span events with flag evaluation information are still added to the parent span.
• include the flag’s evaluated flag value in the span event.

Here’s how:

1
import ldclient
2
from ldclient.config import Config
3
4
from ldotel.tracing import Hook, HookOptions
5
6
config = Config(sdk_key, hooks=[Hook(HookOptions(add_spans=True, include_value=True))])
7
ldclient.set_config(config=config)
8
client = ldclient.get()

To learn more, read Config.

To add flag evaluation information to OpenTelemetry spans, first install the gem:

$
gem install launchdarkly-server-sdk-otel

Then, you can create a new tracing hook and reference it in the configuration options when you initialize the SDK client.

Here’s how:

1
require 'ldclient-otel'
2
3
config = LaunchDarkly::Config.new(hooks: [LaunchDarkly::Otel::TracingHook.new])
4
5
client = LaunchDarkly::LDClient.new("YOUR_SDK_KEY", config);

When this configuration is set, the SDK adds span events for each call to a *Variation* method to your active span. The span event information contains details of each flag evaluation, including the flag key and the context key. You can collect this information in Guarded rollouts or other observability tools.

Optionally, you can configure the tracing hook to:

• create new spans. The new spans are nested in the current active span, and are not active. The span events with flag evaluation information are still added to the parent span.
• include the flag’s evaluated flag value in the span event.

Here’s how:

1
require 'ldclient-otel'
2
3
tracing_hook_options = LaunchDarkly::Otel::TracingHookOptions.new(add_spans: true, include_value: true)
4
hook = LaunchDarkly::Otel::TracingHook.new(tracing_hook_options)
5
config = LaunchDarkly::Config.new(hooks: [hook])
6
7
client = LaunchDarkly::LDClient.new("YOUR_SDK_KEY", config);

To learn more, read Config.