> ## Documentation Index
> Fetch the complete documentation index at: https://launchdarkly.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Instrumenting LLM applications

<View title="Developer" />

<View title="Federal docs" />

<View title="EU docs" />

This topic explains how to set up LLM observability in your SDK and instrument your application so large language model (LLM) spans and conversations render correctly in the LaunchDarkly user interface.

LaunchDarkly builds LLM observability and conversation views from the OpenTelemetry generative AI semantic conventions. If your instrumentation already emits those conventions, LLM observability works without additional configuration. To learn how LaunchDarkly displays the resulting spans and conversations, read [LLM observability](/docs/home/observability/llm-observability).

## Prerequisites

To set up LLM observability, you need:

* Access to LaunchDarkly observability
* A LaunchDarkly [SDK key](/docs/home/account/environment/keys#view-or-copy-sdk-credentials) for your environment

Configuration requirements vary by SDK.

## Set up LLM observability by SDK

When your application calls an LLM provider, instrumentation captures telemetry about the model request, the LaunchDarkly observability SDK exports that telemetry, and LaunchDarkly records the request as an LLM span on the **Traces** page.

### Python

<Accordion title="Expand Python code sample">
  To enable LLM observability in Python:

  1. Install and initialize the [Python SDK](/docs/sdk/server-side/python) with the observability plugin.
  2. Initialize your LLM provider.

  Supported providers do not require additional instrumentation libraries. The Python observability plugin automatically instruments OpenAI, Anthropic, LangChain, LlamaIndex, Amazon Bedrock, and Google Vertex AI, among others. For the full list, read [OpenTelemetry versions and instrumentation support](/docs/sdk/features/observability-otel-versions#python).

  To install the SDK and the observability plugin:

  <CodeGroup>
    ```bash title="Python: Install the SDK and observability plugin" lines wrap theme={null}
    pip install launchdarkly-server-sdk launchdarkly-observability
    ```
  </CodeGroup>

  Then construct the plugin at application startup, before your application calls a model provider. Constructing `ObservabilityPlugin` starts OpenTelemetry auto-instrumentation:

  <CodeGroup>
    ```python title="Python: Initialize the observability plugin" expandable lines wrap theme={null}
    import ldclient
    from ldclient.config import Config
    from ldobserve import ObservabilityConfig, ObservabilityPlugin

    plugin = ObservabilityPlugin(
        ObservabilityConfig(
            service_name="my-llm-app",
            service_version="1.0.0",
        )
    )

    ldclient.set_config(
        Config(
            sdk_key="your-sdk-key",
            plugins=[plugin],
        )
    )
    ```
  </CodeGroup>

  After the plugin initializes, calls to a supported provider generate LLM spans without further changes to your model code:

  <CodeGroup>
    ```python title="Python: Call a supported provider" lines wrap theme={null}
    from openai import OpenAI

    openai_client = OpenAI()

    response = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Summarize this release."}],
    )
    ```
  </CodeGroup>

  To disable instrumentation for a specific library, set `disabled_instrumentations` in `ObservabilityConfig`, or set the `OTEL_PYTHON_DISABLED_INSTRUMENTATIONS` environment variable. For the full set of plugin options, read [Configuration for server-side observability](/docs/sdk/features/observability-config-server-side#python).
</Accordion>

### Node.js

<Accordion title="Expand Node.js code sample">
  To enable LLM observability in Node.js:

  1. Install and initialize the [Node.js (server-side) SDK](/docs/sdk/server-side/node-js).
  2. Register provider instrumentation for your LLM library.
  3. Initialize your LLM provider.

  Initialize the LaunchDarkly SDK before your provider so LaunchDarkly can capture LLM spans correctly. After setup, LaunchDarkly records model calls as LLM spans on the **Traces** page.

  To instrument OpenAI in a Node.js application, install the provider instrumentation package:

  <CodeGroup>
    ```bash title="JavaScript: Install OpenAI instrumentation" lines wrap theme={null}
    npm install --save @traceloop/instrumentation-openai
    ```
  </CodeGroup>

  Then register the instrumentation in your application:

  <CodeGroup>
    ```javascript title="JavaScript: Register OpenAI instrumentation" lines wrap theme={null}
    const { OpenAIInstrumentation } = require("@traceloop/instrumentation-openai");
    const { registerInstrumentations } = require("@opentelemetry/instrumentation");

    // Initialize the LaunchDarkly SDK before registering instrumentations

    registerInstrumentations({
      instrumentations: [new OpenAIInstrumentation()],
    });
    ```
  </CodeGroup>
</Accordion>

### Unsupported providers

If your provider does not support automatic instrumentation, you can manually instrument LLM spans. Manual instrumentation lets you create spans around model calls, attach metadata such as token counts and request duration, and ensure LaunchDarkly associates model activity with LLM spans.

To learn how, read [Manually instrument LLM spans](/docs/home/agentcontrol/manual-llm-span-tracing).

## Instrument your application for conversations

LaunchDarkly normalizes several common non-standard formats when it receives them, including the OpenLLMetry `llm.*` attributes and Claude Code telemetry. If you use one of those, you can rely on the normalized behavior described here.

### Set the conversation ID

Set `gen_ai.conversation.id` on every span that belongs to the conversation, including tool spans, agent spans, and any spans your framework creates in between.

This is the only requirement. Without it, spans never join a conversation and the conversation does not appear in the LaunchDarkly UI.

<Warning>
  **Use the semantic convention key**

  LaunchDarkly groups conversations on `gen_ai.conversation.id` alone. Similar-looking attributes such as `session.id` or `conversation_id` are not used for grouping, because they also appear in session replay data and in backend persistence spans that are not part of the conversation.

  If your framework emits a session identifier under a different key, map it to `gen_ai.conversation.id` in your instrumentation.
</Warning>

### Describe each span

Set these attributes on your model call and tool spans:

<table>
  <thead>
    <tr>
      <th>Attribute</th>
      <th>Purpose</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`gen_ai.operation.name`</td>
      <td>What the span represents. Use a standard value such as `chat`, `execute_tool`, `invoke_agent`, `embeddings`, or `retrieval`. This determines how LaunchDarkly categorizes the span and assigns a matching icon.</td>
    </tr>

    <tr>
      <td>`gen_ai.provider.name`</td>
      <td>The provider handling the request. Required for cost estimation.</td>
    </tr>

    <tr>
      <td>`gen_ai.request.model`</td>
      <td>The model handling the request. Required for cost estimation and for the Model column.</td>
    </tr>

    <tr>
      <td>`gen_ai.usage.input_tokens`</td>
      <td>Input token count.</td>
    </tr>

    <tr>
      <td>`gen_ai.usage.output_tokens`</td>
      <td>Output token count.</td>
    </tr>

    <tr>
      <td>`gen_ai.usage.cache_read.input_tokens`</td>
      <td>Tokens read from a prompt cache. Set this if you use prompt caching.</td>
    </tr>

    <tr>
      <td>`gen_ai.usage.cache_creation.input_tokens`</td>
      <td>Tokens written to a prompt cache. Set this if you use prompt caching.</td>
    </tr>
  </tbody>
</table>

<Warning>
  **Use the current token attribute names**

  Report token usage as `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`. The older `prompt_tokens` and `completion_tokens` names are recognized in the conversation view but not in the Conversations list, so a conversation instrumented with the older names displays token counts in the conversation and zeros in the list.
</Warning>

<Tip>
  **Emit one span per model call**

  Per-turn latency, token counts, and cost come from spans that represent a single inference request. If your agent emits one span for an entire turn, LaunchDarkly still displays the transcript. However, it does not attribute per-call metrics to individual turns.

  If your framework emits an envelope span per turn, emit child spans for each model call underneath it. LaunchDarkly matches the two together automatically.
</Tip>

### Emit message content

Provide conversation message content as `gen_ai.input.messages` and `gen_ai.output.messages`, either as span attributes or as attributes on a `gen_ai.client.inference.operation.details` span event. Use the standard message format:

<CodeGroup>
  ```json title="Message format" lines wrap theme={null}
  [
    {
      "role": "assistant",
      "parts": [
        { "type": "text", "content": "Let me look that up." },
        {
          "type": "tool_call",
          "id": "call_1",
          "name": "search_orders",
          "arguments": "{\"customer_id\":\"1234\"}"
        }
      ]
    }
  ]
  ```
</CodeGroup>

Set `gen_ai.system_instructions` for the system prompt. LaunchDarkly displays it once per conversation, even when your agent re-sends it on every trace.

Emit message deltas where you can. If your agent re-sends the entire running transcript on each request, LaunchDarkly reconstructs the true sequence, but timing for the recovered history is less precise and those turns display without a time offset.

<Note>
  **Content is matched by structure, not by text**

  LaunchDarkly reconciles repeated observations of the same message across many spans. Identical messages that genuinely occur twice, such as a person saying "yes" at two different points, remain separate turns.
</Note>

### Structure tool calls

Tool calls need to connect to the message that requested them. Getting this wrong is the most common cause of a transcript that reads out of order.

Set these attributes on the tool span:

* `gen_ai.tool.name`: The name of the tool that ran.
* `gen_ai.tool.call.id`: The tool call identifier. LaunchDarkly pairs a tool with the model call that requested it by matching this value against the `tool_call` part in that model call's output messages.
* `gen_ai.tool.call.arguments`: The arguments passed to the tool.
* `gen_ai.tool.call.result`: The result the tool returned.

Then include a matching `tool_call` part in the `gen_ai.output.messages` of the model call that requested the tool, using the same ID.

<Warning>
  **Always include the tool\_call part and the call ID**

  These two details do more work than their size suggests, and neither one produces an error when omitted.

  Include a `tool_call` part on the model call that requested the tool. Without it, LaunchDarkly times the assistant's message at the end of its span while timing the tool at its start. Because agent frameworks commonly finish a message only after its tool has run, the tool span often begins before the message ends, and the tool renders before the text that asked for it.

  Set `gen_ai.tool.call.id` to the same value in both places. This is what pairs a tool with its turn. Tool spans emitted as siblings rather than children of the model call have no ancestry relationship to fall back on, so without a matching ID the tool appears on its own. The ID is also what collapses the several spans some frameworks emit for one logical tool call into a single tool card.
</Warning>

<Warning>
  **Keep tool\_call parts off the envelope span**

  If your agent emits a run-level or envelope span alongside per-call spans, put prose on the envelope and causality on the per-call spans. Emit one message per turn on the envelope, and no `tool_call` parts.

  Adding `tool_call` parts to the envelope makes LaunchDarkly order that span's turns relative to the envelope's start rather than to each model call, which collapses every turn to the beginning of the run and flattens the timeline.
</Warning>

### Capture the opening question

A conversation only displays the person's opening question if your instrumentation records it. Always write `gen_ai.input.messages` on the span that receives the request, usually the run or envelope span.

Without this attribute, the conversation transcript opens with the agent answering a question that is not displayed.

Key the text per message rather than accumulating it in a single buffer. Because some frameworks emit the user's prompt as an ordinary text part, a combined output buffer records the person's question as agent output.

### Report evaluations

To attach a quality score to a turn, emit a `gen_ai.evaluation.result` span event on the span being evaluated. Use these attributes in the event:

* `gen_ai.evaluation.name`: The name of the measure, such as `relevance` or `toxicity`.
* `gen_ai.evaluation.score.value`: A numeric score.
* `gen_ai.evaluation.score.label`: A text label. LaunchDarkly uses this label in place of the numeric score when it is present.
* `gen_ai.evaluation.explanation`: Optional reasoning, displayed on hover.

Attach evaluation results to the span where you want them to appear. A result that is attached to a separate evaluation span appears on that span rather than on the turn it describes.

### Avoid a common misclassification

<Warning>
  **Do not include HTTP or database attributes in inference spans**

  LaunchDarkly classifies each span by a single operation type, and HTTP, GraphQL, messaging, and database attributes all take priority over generative AI attributes.

  A span that carries both HTTP and generative AI attributes is classified as an HTTP span. It still appears in the conversation view, which reads the attributes directly. However, it drops out of the [LLM Agent default dashboard](/docs/home/observability/dashboards#llm-agent-default-dashboard) and the LLM spans filter, because both select by operation type. This is the most frequent reason that a dashboard might report less activity than an agent actually produced.

  Record the outbound HTTP call on a child span instead, and keep the inference span to generative AI attributes.
</Warning>