Instrumenting LLM applications

This topic explains how to set up LLM observability in your SDK and instrument your application so large language model (LLM) spans and conversations render correctly in the LaunchDarkly user interface.

LaunchDarkly builds LLM observability and conversation views from the OpenTelemetry generative AI semantic conventions. If your instrumentation already emits those conventions, LLM observability works without additional configuration. To learn how LaunchDarkly displays the resulting spans and conversations, read LLM observability.

Prerequisites

To set up LLM observability, you need:

• Access to LaunchDarkly observability
• A LaunchDarkly SDK key for your environment

Configuration requirements vary by SDK.

Set up LLM observability by SDK

When your application calls an LLM provider, instrumentation captures telemetry about the model request, the LaunchDarkly observability SDK exports that telemetry, and LaunchDarkly records the request as an LLM span on the Traces page.

Python

$
pip install launchdarkly-server-sdk launchdarkly-observability

1
import ldclient
2
from ldclient.config import Config
3
from ldobserve import ObservabilityConfig, ObservabilityPlugin
4
5
plugin = ObservabilityPlugin(
6
    ObservabilityConfig(
7
        service_name="my-llm-app",
8
        service_version="1.0.0",
9
    )
10
)
11
12
ldclient.set_config(
13
    Config(
14
        sdk_key="your-sdk-key",
15
        plugins=[plugin],
16
    )
17
)

1
from openai import OpenAI
2
3
openai_client = OpenAI()
4
5
response = openai_client.chat.completions.create(
6
    model="gpt-4o",
7
    messages=[{"role": "user", "content": "Summarize this release."}],
8
)

Node.js

$
npm install --save @traceloop/instrumentation-openai

1
const { OpenAIInstrumentation } = require("@traceloop/instrumentation-openai");
2
const { registerInstrumentations } = require("@opentelemetry/instrumentation");
3
4
// Initialize the LaunchDarkly SDK before registering instrumentations
5
6
registerInstrumentations({
7
  instrumentations: [new OpenAIInstrumentation()],
8
});

Unsupported providers

If your provider does not support automatic instrumentation, you can manually instrument LLM spans. Manual instrumentation lets you create spans around model calls, attach metadata such as token counts and request duration, and ensure LaunchDarkly associates model activity with LLM spans.

To learn how, read Manually instrument LLM spans.

Instrument your application for conversations

LaunchDarkly normalizes several common non-standard formats when it receives them, including the OpenLLMetry llm.* attributes and Claude Code telemetry. If you use one of those, you can rely on the normalized behavior described here.

Set the conversation ID

Set gen_ai.conversation.id on every span that belongs to the conversation, including tool spans, agent spans, and any spans your framework creates in between.

This is the only requirement. Without it, spans never join a conversation and the conversation does not appear in the LaunchDarkly UI.

Describe each span

Set these attributes on your model call and tool spans:

Emit message content

Provide conversation message content as gen_ai.input.messages and gen_ai.output.messages, either as span attributes or as attributes on a gen_ai.client.inference.operation.details span event. Use the standard message format:

1
[
2
  {
3
    "role": "assistant",
4
    "parts": [
5
      { "type": "text", "content": "Let me look that up." },
6
      {
7
        "type": "tool_call",
8
        "id": "call_1",
9
        "name": "search_orders",
10
        "arguments": "{\"customer_id\":\"1234\"}"
11
      }
12
    ]
13
  }
14
]

Set gen_ai.system_instructions for the system prompt. LaunchDarkly displays it once per conversation, even when your agent re-sends it on every trace.

Emit message deltas where you can. If your agent re-sends the entire running transcript on each request, LaunchDarkly reconstructs the true sequence, but timing for the recovered history is less precise and those turns display without a time offset.

Structure tool calls

Tool calls need to connect to the message that requested them. Getting this wrong is the most common cause of a transcript that reads out of order.

Set these attributes on the tool span:

• gen_ai.tool.name: The name of the tool that ran.
• gen_ai.tool.call.id: The tool call identifier. LaunchDarkly pairs a tool with the model call that requested it by matching this value against the tool_call part in that model call’s output messages.
• gen_ai.tool.call.arguments: The arguments passed to the tool.
• gen_ai.tool.call.result: The result the tool returned.

Then include a matching tool_call part in the gen_ai.output.messages of the model call that requested the tool, using the same ID.

Capture the opening question

A conversation only displays the person’s opening question if your instrumentation records it. Always write gen_ai.input.messages on the span that receives the request, usually the run or envelope span.

Without this attribute, the conversation transcript opens with the agent answering a question that is not displayed.

Key the text per message rather than accumulating it in a single buffer. Because some frameworks emit the user’s prompt as an ordinary text part, a combined output buffer records the person’s question as agent output.

Report evaluations

To attach a quality score to a turn, emit a gen_ai.evaluation.result span event on the span being evaluated. Use these attributes in the event:

• gen_ai.evaluation.name: The name of the measure, such as relevance or toxicity.
• gen_ai.evaluation.score.value: A numeric score.
• gen_ai.evaluation.score.label: A text label. LaunchDarkly uses this label in place of the numeric score when it is present.
• gen_ai.evaluation.explanation: Optional reasoning, displayed on hover.

Attach evaluation results to the span where you want them to appear. A result that is attached to a separate evaluation span appears on that span rather than on the turn it describes.

Avoid a common misclassification