Instrumenting LLM applications

This topic explains how to set up LLM observability in your SDK and instrument your application so large language model (LLM) spans and conversations render correctly in the LaunchDarkly user interface. LaunchDarkly builds LLM observability and conversation views from the OpenTelemetry generative AI semantic conventions. If your instrumentation already emits those conventions, LLM observability works without additional configuration. To learn how LaunchDarkly displays the resulting spans and conversations, read LLM observability.

Prerequisites

To set up LLM observability, you need:

Access to LaunchDarkly observability
A LaunchDarkly SDK key for your environment

Configuration requirements vary by SDK.

Set up LLM observability by SDK

When your application calls an LLM provider, instrumentation captures telemetry about the model request, the LaunchDarkly observability SDK exports that telemetry, and LaunchDarkly records the request as an LLM span on the Traces page.

Python

Expand Python code sample

To enable LLM observability in Python:

Install and initialize the Python SDK with the observability plugin.
Initialize your LLM provider.

Supported providers do not require additional instrumentation libraries. The Python observability plugin automatically instruments OpenAI, Anthropic, LangChain, LlamaIndex, Amazon Bedrock, and Google Vertex AI, among others. For the full list, read OpenTelemetry versions and instrumentation support.To install the SDK and the observability plugin:

pip install launchdarkly-server-sdk launchdarkly-observability

Then construct the plugin at application startup, before your application calls a model provider. Constructing ObservabilityPlugin starts OpenTelemetry auto-instrumentation:

import ldclient
from ldclient.config import Config
from ldobserve import ObservabilityConfig, ObservabilityPlugin

plugin = ObservabilityPlugin(
    ObservabilityConfig(
        service_name="my-llm-app",
        service_version="1.0.0",
    )
)

ldclient.set_config(
    Config(
        sdk_key="your-sdk-key",
        plugins=[plugin],
    )
)

After the plugin initializes, calls to a supported provider generate LLM spans without further changes to your model code:

from openai import OpenAI

openai_client = OpenAI()

response = openai_client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Summarize this release."}],
)

To disable instrumentation for a specific library, set disabled_instrumentations in ObservabilityConfig, or set the OTEL_PYTHON_DISABLED_INSTRUMENTATIONS environment variable. For the full set of plugin options, read Configuration for server-side observability.

Node.js

Expand Node.js code sample

To enable LLM observability in Node.js:

Install and initialize the Node.js (server-side) SDK.
Register provider instrumentation for your LLM library.
Initialize your LLM provider.

Initialize the LaunchDarkly SDK before your provider so LaunchDarkly can capture LLM spans correctly. After setup, LaunchDarkly records model calls as LLM spans on the Traces page.To instrument OpenAI in a Node.js application, install the provider instrumentation package:

npm install --save @traceloop/instrumentation-openai

Then register the instrumentation in your application:

const { OpenAIInstrumentation } = require("@traceloop/instrumentation-openai");
const { registerInstrumentations } = require("@opentelemetry/instrumentation");

// Initialize the LaunchDarkly SDK before registering instrumentations

registerInstrumentations({
  instrumentations: [new OpenAIInstrumentation()],
});

Unsupported providers

If your provider does not support automatic instrumentation, you can manually instrument LLM spans. Manual instrumentation lets you create spans around model calls, attach metadata such as token counts and request duration, and ensure LaunchDarkly associates model activity with LLM spans. To learn how, read Manually instrument LLM spans.

Instrument your application for conversations

LaunchDarkly normalizes several common non-standard formats when it receives them, including the OpenLLMetry llm.* attributes and Claude Code telemetry. If you use one of those, you can rely on the normalized behavior described here.

Set the conversation ID

Set gen_ai.conversation.id on every span that belongs to the conversation, including tool spans, agent spans, and any spans your framework creates in between. This is the only requirement. Without it, spans never join a conversation and the conversation does not appear in the LaunchDarkly UI.

Use the semantic convention keyLaunchDarkly groups conversations on gen_ai.conversation.id alone. Similar-looking attributes such as session.id or conversation_id are not used for grouping, because they also appear in session replay data and in backend persistence spans that are not part of the conversation.If your framework emits a session identifier under a different key, map it to gen_ai.conversation.id in your instrumentation.

Describe each span

Set these attributes on your model call and tool spans:

Attribute	Purpose
`gen_ai.operation.name`	What the span represents. Use a standard value such as `chat`, `execute_tool`, `invoke_agent`, `embeddings`, or `retrieval`. This determines how LaunchDarkly categorizes the span and assigns a matching icon.
`gen_ai.provider.name`	The provider handling the request. Required for cost estimation.
`gen_ai.request.model`	The model handling the request. Required for cost estimation and for the Model column.
`gen_ai.usage.input_tokens`	Input token count.
`gen_ai.usage.output_tokens`	Output token count.
`gen_ai.usage.cache_read.input_tokens`	Tokens read from a prompt cache. Set this if you use prompt caching.
`gen_ai.usage.cache_creation.input_tokens`	Tokens written to a prompt cache. Set this if you use prompt caching.

Use the current token attribute namesReport token usage as gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. The older prompt_tokens and completion_tokens names are recognized in the conversation view but not in the Conversations list, so a conversation instrumented with the older names displays token counts in the conversation and zeros in the list.

Emit one span per model callPer-turn latency, token counts, and cost come from spans that represent a single inference request. If your agent emits one span for an entire turn, LaunchDarkly still displays the transcript. However, it does not attribute per-call metrics to individual turns.If your framework emits an envelope span per turn, emit child spans for each model call underneath it. LaunchDarkly matches the two together automatically.

Emit message content

Provide conversation message content as gen_ai.input.messages and gen_ai.output.messages, either as span attributes or as attributes on a gen_ai.client.inference.operation.details span event. Use the standard message format:

[
  {
    "role": "assistant",
    "parts": [
      { "type": "text", "content": "Let me look that up." },
      {
        "type": "tool_call",
        "id": "call_1",
        "name": "search_orders",
        "arguments": "{\"customer_id\":\"1234\"}"
      }
    ]
  }
]

Set gen_ai.system_instructions for the system prompt. LaunchDarkly displays it once per conversation, even when your agent re-sends it on every trace. Emit message deltas where you can. If your agent re-sends the entire running transcript on each request, LaunchDarkly reconstructs the true sequence, but timing for the recovered history is less precise and those turns display without a time offset.

Content is matched by structure, not by textLaunchDarkly reconciles repeated observations of the same message across many spans. Identical messages that genuinely occur twice, such as a person saying “yes” at two different points, remain separate turns.

Structure tool calls

Tool calls need to connect to the message that requested them. Getting this wrong is the most common cause of a transcript that reads out of order. Set these attributes on the tool span:

gen_ai.tool.name: The name of the tool that ran.
gen_ai.tool.call.id: The tool call identifier. LaunchDarkly pairs a tool with the model call that requested it by matching this value against the tool_call part in that model call’s output messages.
gen_ai.tool.call.arguments: The arguments passed to the tool.
gen_ai.tool.call.result: The result the tool returned.

Then include a matching tool_call part in the gen_ai.output.messages of the model call that requested the tool, using the same ID.

Always include the tool_call part and the call IDThese two details do more work than their size suggests, and neither one produces an error when omitted.Include a tool_call part on the model call that requested the tool. Without it, LaunchDarkly times the assistant’s message at the end of its span while timing the tool at its start. Because agent frameworks commonly finish a message only after its tool has run, the tool span often begins before the message ends, and the tool renders before the text that asked for it.Set gen_ai.tool.call.id to the same value in both places. This is what pairs a tool with its turn. Tool spans emitted as siblings rather than children of the model call have no ancestry relationship to fall back on, so without a matching ID the tool appears on its own. The ID is also what collapses the several spans some frameworks emit for one logical tool call into a single tool card.

Keep tool_call parts off the envelope spanIf your agent emits a run-level or envelope span alongside per-call spans, put prose on the envelope and causality on the per-call spans. Emit one message per turn on the envelope, and no tool_call parts.Adding tool_call parts to the envelope makes LaunchDarkly order that span’s turns relative to the envelope’s start rather than to each model call, which collapses every turn to the beginning of the run and flattens the timeline.

Capture the opening question

A conversation only displays the person’s opening question if your instrumentation records it. Always write gen_ai.input.messages on the span that receives the request, usually the run or envelope span. Without this attribute, the conversation transcript opens with the agent answering a question that is not displayed. Key the text per message rather than accumulating it in a single buffer. Because some frameworks emit the user’s prompt as an ordinary text part, a combined output buffer records the person’s question as agent output.

Report evaluations

To attach a quality score to a turn, emit a gen_ai.evaluation.result span event on the span being evaluated. Use these attributes in the event:

gen_ai.evaluation.name: The name of the measure, such as relevance or toxicity.
gen_ai.evaluation.score.value: A numeric score.
gen_ai.evaluation.score.label: A text label. LaunchDarkly uses this label in place of the numeric score when it is present.
gen_ai.evaluation.explanation: Optional reasoning, displayed on hover.

Attach evaluation results to the span where you want them to appear. A result that is attached to a separate evaluation span appears on that span rather than on the turn it describes.

Avoid a common misclassification

Do not include HTTP or database attributes in inference spansLaunchDarkly classifies each span by a single operation type, and HTTP, GraphQL, messaging, and database attributes all take priority over generative AI attributes.A span that carries both HTTP and generative AI attributes is classified as an HTTP span. It still appears in the conversation view, which reads the attributes directly. However, it drops out of the LLM Agent default dashboard and the LLM spans filter, because both select by operation type. This is the most frequent reason that a dashboard might report less activity than an agent actually produced.Record the outbound HTTP call on a child span instead, and keep the inference span to generative AI attributes.

​Prerequisites

​Set up LLM observability by SDK

​Python

​Node.js

​Unsupported providers

​Instrument your application for conversations

​Set the conversation ID

​Describe each span

​Emit message content

​Structure tool calls

​Capture the opening question

​Report evaluations

​Avoid a common misclassification

Prerequisites

Set up LLM observability by SDK

Python

Node.js

Unsupported providers

Instrument your application for conversations

Set the conversation ID

Describe each span

Emit message content

Structure tool calls

Capture the opening question

Report evaluations

Avoid a common misclassification