> ## Documentation Index
> Fetch the complete documentation index at: https://launchdarkly.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Tracking AI metrics

<View title="Developer" />

<View title="Federal docs" />

<View title="EU docs" />

This topic explains how to record metrics from your AI model generation, including duration, generation, satisfaction, and several token-related metrics.

Use LaunchDarkly AI SDKs to record these metrics by wrapping your AI model provider call with a tracking method so metrics are captured as part of generation. LaunchDarkly displays recorded metrics on the [config's **Monitoring** tab](/docs/home/agentcontrol/monitor).

This feature is available for AI SDKs only. LaunchDarkly AI SDKs are designed for use with LaunchDarkly AgentControl and are currently in a pre-1.0 release under active development.

All AI SDKs include `track*` methods to record:

* duration
* token usage
* generation success
* generation error
* time to first token
* output satisfaction

Each AI SDK also includes a method to retrieve a summary of the metrics that have been automatically collected.

Some AI SDKs also include provider-specific `track_[model]_metrics` methods for configs in completion mode. These methods take the result of a provider call and record:

* duration
* token usage
* generation success
* generation error

You can use provider-specific methods as a shorthand, or call `track*` methods directly to record additional metrics.

Both `track*` and `track_[model]_metrics` methods are called from a `tracker`. The `tracker` is returned by a config [customization call](/docs/sdk/features/agentcontrol-config) and is available for configs in both completion mode and agent mode.

## Record delayed feedback events

The SDK expects you to use the `tracker` within the same request lifecycle that generates AI content. Do not reuse a `tracker` across separate requests.

If user feedback arrives later, you must persist the tracking metadata to ensure the feedback is attributed to the same config variation that generated the content. The `tracker` exposes `getTrackData()` so you can capture this metadata at generation time and reuse it when feedback arrives.

Use the following pattern to record delayed feedback:

1. Use `tracker.getTrackData()` to capture the tracker metadata at generation time.
2. Store the metadata alongside the context where generation results are stored.
3. Retrieve the stored metadata when feedback arrives.
4. Send the tracking event with the original metadata.

Here is an example using `ldClient.track`:

```ts expandable lines wrap theme={null}
// Generation time
const aiConfig = await aiClient.completionConfig(
  aiConfigKey,
  context,
  defaultValue,
  variables
);
const trackData = aiConfig.tracker.getTrackData();

// Persist trackData together with the context

// Feedback time
ldClient.track(
  feedback.kind === "positive"
    ? "$ld:ai:feedback:user:positive"
    : "$ld:ai:feedback:user:negative",
  context, // context from generation time
  trackData,
  1
);
```

## AI SDKs

This feature is available for the following AI SDKs:

* [.NET AI](#net-ai)
* [Go AI](#go-ai)
* [Node.js (server-side) AI](#nodejs-server-side-ai)
* [Node.js (server-side) AI (turnkey)](#nodejs-server-side-ai-turnkey)
* [Python AI](#python-ai)
* [Ruby AI](#ruby-ai)

### .NET AI

<Accordion title="Expand .NET AI SDK code sample">
  Use the `TrackRequest` function to wrap your AI model provider call and record metrics when the model generates content.

  The `tracker` is returned from your call to [customize the config](/docs/sdk/features/agentcontrol-config#net-ai) and is specific to that config. Make sure to call `Config` again each time you generate content from your AI model so that metrics are correctly associated with the customized config variation.

  Here's how to call an AI model provider and record metrics from generation:

  <CodeGroup>
    ```csharp title=".NET AI SDK, any model" expandable lines wrap theme={null}
    if (tracker.Config.Enabled == true) {

      // Use Task.Run here only as an example. You can wrap any async provider call.
      var response = tracker.TrackRequest(Task.Run(() =>
        {
          // Make request to a provider, which automatically tracks metrics in LaunchDarkly.
          // When sending the request to a provider, use details from tracker.Config.
          // For example, you can pass tracker.Config.Model and tracker.Config.Messages.
          // Optionally, return response metadata for additional logging.
          //
          // CAUTION: If the call inside Task.Run throws an exception,
          // the SDK will re-throw that exception.

          return new Response
          {
            Usage = new Usage { Total = 1, Input = 1, Output = 1 }, // Token usage data
            Metrics = new Metrics { LatencyMs = 100 } // Metrics data
          };
        }
      ));

    } else {

      // Application path to take when tracker.Config is disabled

    }
    ```
  </CodeGroup>

  If you want to perform additional tracking beyond what LaunchDarkly provides automatically, populate the `Response` object with the metrics you want to record.

  You can use the SDK's other `Track*` functions to record metrics manually. The `TrackRequest` function expects a response, so manual tracking may be required for streaming use cases.

  Each of the `Track*` functions sends data back to LaunchDarkly. The [Monitoring tab](/docs/home/agentcontrol/monitor) of the config in the LaunchDarkly UI aggregates metrics from all variations of the config.

  Here's how to record metrics manually:

  <CodeGroup>
    ```csharp title="Track duration" lines wrap theme={null}
    /// Track your own start and stop time.
    /// Set duration to the time (in ms) that your AI model generation takes.
    /// The duration may include network latency, depending on how you calculate it.

    tracker.TrackDuration(response.Metrics.LatencyMs);
    ```

    ```csharp title="Track token usage" lines wrap theme={null}
    /// Track your own token usage.

    tracker.TrackTokens(response.Usage);
    ```

    ```csharp title="Track output satisfaction rate" lines wrap theme={null}
    /// Track your own output satisfaction rate.

    /// Pass in Feedback.Positive or Feedback.Negative.
    tracker.TrackFeedback(Feedback.Positive);
    ```

    ```csharp title="Track generation (success)" lines wrap theme={null}
    /// Track a successful generation event using cfg

    tracker.TrackSuccess();
    ```

    ```csharp title="Track generation (error)" lines wrap theme={null}
    /// Track an unsuccessful generation event using cfg

    tracker.TrackError();
    ```

    ```csharp title="Track time to first token" lines wrap theme={null}
    /// Track the time it takes to generate the first token
    /// Pass in the time (in ms) until your first token is generated
    /// This may include network latency, depending on how you calculate it
    tracker.TrackTimeToFirstToken(1000);
    ```
  </CodeGroup>

  The SDK automatically flushes pending analytics events to LaunchDarkly at regular intervals. If you have a short-lived application, you may need to explicitly request that the underlying LaunchDarkly client deliver any pending analytics events to LaunchDarkly, using [`flush()`](/docs/sdk/features/flush) or [`close()`](/docs/sdk/features/shutdown).

  Here's how:

  <CodeGroup>
    ```csharp title="Flush tracking events" lines wrap theme={null}
    baseClient.Flush();
    ```
  </CodeGroup>

  To learn more, read [`LDAIConfigTracker`](https://launchdarkly.github.io/dotnet-core/pkgs/sdk/server-ai/api/LaunchDarkly.Sdk.Server.Ai.LdAiConfigTracker.html).
</Accordion>

### Go AI

<Accordion title="Expand Go AI SDK code sample">
  Use the `TrackRequest()` function to wrap your AI model provider call and record metrics from generation.

  The `tracker` is returned from your call to [customize the config](/docs/sdk/features/agentcontrol-config#go-ai) and is specific to that config. Make sure to call `Config()` again each time you use the tracker and generate content from your AI model so that your metrics are correctly associated with the customized config variation.

  Here's how to call an AI model provider and record metrics from generation:

  <CodeGroup>
    ```go title="Go AI SDK, any model" expandable lines wrap theme={null}
    if cfg.Enabled() {

      response, err := tracker.TrackRequest(func(config *Config) (ProviderResponse, error) {

        // Make request to a provider, which automatically tracks metrics in LaunchDarkly.
        // When sending the request to a provider, use details from config.
        // For example, you can pass a model parameter (config.ModelParam) or messages (config.Messages).
        // Optionally, return response metadata for additional logging.

        return ProviderResponse{
          Usage: TokenUsage{
            Total: 1, // Token usage data
          },
          Metrics: Metrics{
            Latency: 10 * time.Millisecond, // Metrics data
          },
        }, nil
      })

    } else {

      // Application path to take when the configuration is disabled

    }
    ```
  </CodeGroup>

  You can use the SDK's other `Track*` functions to record metrics manually. The `TrackRequest` function expects a response, so manual tracking may be required for streaming use cases.

  Each `Track*` function sends data back to LaunchDarkly. To review the metrics that have been recorded, use `GetSummary`:

  <CodeGroup>
    ```go title="Get automatically recorded metrics" lines wrap theme={null}
    summary := tracker.GetSummary();

    // recorded metrics available in summary.Duration, summary.Feedback,
    // summary.Tokens, summary.Success, summary.TimeToFirstToken

    ```
  </CodeGroup>

  To learn more, read [`GetSummary`](https://pkg.go.dev/github.com/launchdarkly/go-server-sdk-ai/ldai#Tracker.GetSummary) and [`MetricSummary`](https://pkg.go.dev/github.com/launchdarkly/go-server-sdk-ai/ldai#MetricSummary).

  The [Monitoring tab](/docs/home/agentcontrol/monitor) of the config in the LaunchDarkly UI aggregates data from the `Track*` functions from across all variations of the config.

  Here's how to record metrics manually:

  <CodeGroup>
    ```go title="Track duration" lines wrap theme={null}
    // Track your own start and stop time.

    // Set duration to the time that your AI model generation takes.
    // The duration may include network latency, depending on how you calculate it.

    tracker.TrackDuration(10 * time.Millisecond);
    ```

    ```go title="Track token usage" lines wrap theme={null}
    // Track your own token usage.

    tracker.TrackTokens(response.Usage)
    ```

    ```go title="Track output satisfaction rate" lines wrap theme={null}
    // Track your own output satisfaction rate.

    // Pass in feedbackPositive or feedbackNegative.
    tracker.TrackFeedback(ldai.FeedbackPositive);
    ```

    ```go title="Track generation (success)" lines wrap theme={null}
    // Track a successful generation event using cfg

    tracker.TrackSuccess();
    ```

    ```go title="Track generation (error)" lines wrap theme={null}
    // Track an unsuccessful generation event using cfg

    tracker.TrackError();
    ```

    ```go title="Track time to first token" lines wrap theme={null}
    // Track the time it takes to generate the first token

    // Pass in the time (in ms) until your first token is generated
    // This may include network latency, depending on how you calculate it

    tracker.TrackTimeToFirstToken(10 * time.Millisecond);
    ```
  </CodeGroup>

  The SDK automatically flushes these pending analytics events to LaunchDarkly at regular intervals. If you have a short-lived application, you may need to explicitly request that the underlying LaunchDarkly client deliver any pending analytics events to LaunchDarkly, using [`flush()`](/docs/sdk/features/flush) or [`close()`](/docs/sdk/features/shutdown).

  Here's how:

  <CodeGroup>
    ```go title="Flush tracking events" lines wrap theme={null}
    client.Flush();
    ```
  </CodeGroup>

  To learn more, read [`Tracker`](https://pkg.go.dev/github.com/launchdarkly/go-server-sdk-ai/ldai#Tracker).
</Accordion>

### Java AI

<Accordion title="Expand Java AI SDK code sample">
  Use the tracker's `trackDurationOf` method to wrap your AI model provider call and record metrics from generation.

  The tracker is returned from your call to customize the config (`completionConfig`, `agentConfig`, or `agentConfigs`) and is specific to that config. Make sure to call the config method again each time you generate content from your AI model, so that your metrics are correctly associated with the customized config variation.

  Here's how to call an AI model provider and record metrics from generation:

  <CodeGroup>
    ```java title="Java AI SDK, any model" lines wrap theme={null}
    if (config.isEnabled()) {

      LDAIConfigTracker tracker = config.createTracker();

      Response response = tracker.trackDurationOf(() -> {
        // Make a request to your generative AI provider, using details from
        // the config. For example, pass config.getModel() and config.getMessages().
        return callProvider();
      });

      tracker.trackTokens(new TokenUsage(response.getTotalTokens(), response.getInputTokens(), response.getOutputTokens()));
      tracker.trackSuccess();

    } else {

      // Application path to take when the config is disabled

    }
    ```
  </CodeGroup>

  You can use the SDK's other `track*` methods to record metrics manually. `trackDurationOf` expects a return value, so manual tracking may be required for streaming use cases.

  Each `track*` method sends data back to LaunchDarkly. To review the metrics that have been recorded, use `getSummary`:

  <CodeGroup>
    ```java title="Get automatically recorded metrics" lines wrap theme={null}
    MetricSummary summary = tracker.getSummary();

    // Recorded metrics are available in: 
    // getSuccess() -> Boolean
    // getTokens() -> TokenUsage
    // getDurationMs() -> Long
    // getFeedback() -> FeedbackKind
    // getTimeToFirstTokenMs() -> Long
    // getToolCalls() -> List<String>
    // getResumptionToken() -> String
    ```
  </CodeGroup>

  The **Monitoring** tab of the config in the LaunchDarkly UI aggregates data from the `track*` methods across all variations of the config.

  Here's how to record metrics manually:

  <CodeGroup>
    ```java title="Track duration" lines wrap theme={null}
    // Track your own start and stop time.
    // Set duration to the time that your AI model generation takes.
    // The duration may include network latency, depending on how you calculate it.

    tracker.trackDuration(Duration.ofMillis(100));
    ```

    ```java title="Track token usage" lines wrap theme={null}
    tracker.trackTokens(new TokenUsage(totalTokens, inputTokens, outputTokens));
    ```

    ```java title="Track output satisfaction rate" lines wrap theme={null}
    tracker.trackFeedback(FeedbackKind.POSITIVE);
    ```

    ```java title="Track generation (success)" lines wrap theme={null}
    tracker.trackSuccess();
    ```

    ```java title="Track generation (error)" lines wrap theme={null}
    tracker.trackError();
    ```

    ```java title="Track time to first token" lines wrap theme={null}
    tracker.trackTimeToFirstToken(Duration.ofMillis(50));
    ```
  </CodeGroup>

  The SDK automatically flushes these pending analytics events to LaunchDarkly at regular intervals. If you have a short-lived application, you may need to explicitly request that the underlying LaunchDarkly client deliver any pending analytics events to LaunchDarkly, using `flush()` or `close()`.

  Here's how:

  <CodeGroup>
    ```java title="Flush tracking events" lines wrap theme={null}
    ldClient.flush();
    ```
  </CodeGroup>

  To learn more, read [`LDAIConfigTracker`](https://launchdarkly.github.io/java-core/lib/sdk/server-ai/).
</Accordion>

### Node.js (server-side) AI

<Accordion title="Expand Node.js (server-side) AI SDK code sample">
  If your config uses completion mode, the Node.js (server-side) AI SDK provides several options for making a request to your generative AI provider and recording metrics from your AI model generation. You can use any of the following options:

  * If you are working with OpenAI or Bedrock Converse, use the `trackOpenAIMetrics` or `trackBedrockConverseMetrics` functions, respectively, to record metrics. These functions take the result of your generative operation as a parameter.
  * If you are working with Vercel, use either of the `trackVercelAISDKGenerateTextMetrics` or `trackVercelAISDKStreamTextMetrics` functions to record metrics. These functions take the result of a generative operation from any provider supported by the Vercel AI SDK as a parameter.
  * If you are using a generative AI provider or framework for which the SDK does not provide a convenience function, use the SDK's other `track*` functions to record metrics manually.

  If your config uses `agent` mode, you can access the `instructions` returned from the `agentConfig()` call to send to your AI model. Use the `tracker` returned in this call to record metrics.

  In the following examples, the `tracker` is from your call to [customize the config](/docs/sdk/features/agentcontrol-config#nodejs-server-side-ai), and is specific to that config. Make sure to call `completionConfig` again each time you use the tracker and generate content from your AI model, so that your metrics are correctly associated with the customized config variation.

  Here's how:

  <CodeGroup>
    ```ts title="Using OpenAI model, completion mode" maxLines=0 expandable lines wrap theme={null}
    const { tracker } = aiConfig;

    if (aiConfig.enabled) {

      // Pass in the result of the OpenAI operation.
      // When you call the OpenAI operation, use details from aiConfig.
      // For instance, you can pass aiConfig.messages
      // and aiConfig.model to your specific OpenAI operation.
      //
      // CAUTION: If the call inside of trackOpenAIMetrics throws an exception,
      // the SDK will re-throw that exception

      const completion = await tracker.trackOpenAIMetrics(async () =>
        client.chat.completions.create({
          messages: aiConfig.messages || [],
          model: aiConfig.model?.name || 'gpt-4',
          temperature: (aiConfig.model?.parameters?.temperature as number) ?? 0.5,
          maxTokens: (aiConfig.model?.parameters?.maxTokens as number) ?? 4096,
        }),
      );

    } else {

      // Application path to take when the aiConfig is disabled

    }
    ```

    ```ts title="Using Bedrock model, completion mode" maxLines=0 expandable lines wrap theme={null}
    const { tracker } = aiConfig;

    if (aiConfig.enabled) {

      // Pass in the result of the Bedrock Converse command.
      // When you call the Bedrock Converse command, use details from aiConfig.
      // For instance, you can pass aiConfig.messages
      // and aiConfig.model to your specific Bedrock Converse command.

      const completion = tracker.trackBedrockConverseMetrics(
        await awsClient.send(
          new ConverseCommand({
            modelId: aiConfig.model?.name ?? 'no-model',
            messages: mapPromptToConversation(aiConfig.messages ?? []),
            inferenceConfig: {
              temperature: (aiConfig.model?.parameters?.temperature as number) ?? 0.5,
              maxTokens: (aiConfig.model?.parameters?.maxTokens as number) ?? 4096,
            },
          }),
        ),
      );

    } else {

      // Application path to take when the aiConfig is disabled

    }
    ```

    ```js title="Accessing instructions and recording metrics, agent mode" lines wrap theme={null}
    if (agent.enabled) {

      // Retrieve instructions from the config and pass to your AI model
      const result = example_model_api(agent.instructions)

      // Track metrics from the result
      agent.tracker.trackSuccess()

    } else {

      // Application path to take when the agent config is disabled

    }
    ```
  </CodeGroup>

  Here's how to make a request using the Vercel AI SDK's [`generateText`](https://ai-sdk.dev/docs/ai-sdk-core/generating-text#generatetext) or [`streamText`](https://ai-sdk.dev/docs/ai-sdk-core/generating-text#streamtext), and record the metrics:

  <CodeGroup>
    ```ts title="Using Vercel and generateText" maxLines=0 lines wrap theme={null}
    const { tracker } = aiConfig;

    // Pass in the result of the Vercel AI SDK's generateText function.
    // When you call generateText, use details from the aiConfig,
    // mapped to the input format required for the Vercel AI SDK.
    //
    // CAUTION: The toVercelAISDK function may throw an exception
    // if a Vercel AI SDK model cannot be determined.

    const completion = await tracker.trackVercelAISDKGenerateTextMetrics(() =>
     generateText(
        aiConfig.toVercelAISDK(vercelProvider, vercelProviderOptions)
      )
    )

    ```

    ```ts title="Using Vercel and streamText" maxLines=0 lines wrap theme={null}
    const { tracker } = aiConfig;

    // Pass in the result of the Vercel AI SDK's streamText function.
    // When you call streamText, use details from the aiConfig,
    // mapped to the input format required for the Vercel AI SDK.
    //
    // CAUTION: The toVercelAISDK function may throw an exception
    // if a Vercel AI SDK model cannot be determined.

    const completion = tracker.trackVercelAISDKStreamTextMetrics(() =>
     streamText(
        aiConfig.toVercelAISDK(vercelProvider, vercelProviderOptions)
      )
    )

    ```
  </CodeGroup>

  To learn more, read [`trackVercelAISDKGenerateTextMetrics`](https://launchdarkly.github.io/js-core/packages/sdk/server-ai/docs/interfaces/LDAIConfigTracker.html#trackVercelAISDKGenerateTextMetrics), [`trackVercelAISDKStreamTextMetrics`](https://launchdarkly.github.io/js-core/packages/sdk/server-ai/docs/interfaces/LDAIConfigTracker.html#trackVercelAISDKStreamTextMetrics), and [`toVercelAISDK`](https://launchdarkly.github.io/js-core/packages/sdk/server-ai/docs/interfaces/LDAIConfig.html#toVercelAISDK).

  You can use the SDK's other `track*` functions to record these metrics manually. You may need to do this if you are using a model for which the SDK does not provide a convenience `track[Model]Metrics` function, and you are not using the Vercel AI SDK. The `track[Model]Metrics` functions are expecting a response, so you may also need to do this if your application requires streaming.

  Each of the `track*` functions sends data back to LaunchDarkly. To review the metrics that have been recorded, use `getSummary`:

  <CodeGroup>
    ```ts title="Get automatically recorded metrics, completion mode" lines wrap theme={null}
    const summary = aiConfig.tracker.getSummary();

    // recorded metrics available in summary.durationMs, summary.feedback,
    // summary.tokens, summary.success, summary.timeToFirstTokenMs

    ```

    ```ts title="Get automatically recorded metrics, agent mode" lines wrap theme={null}
    const summary = agent.tracker.getSummary();

    // recorded metrics available in summary.durationMs, summary.feedback,
    // summary.tokens, summary.success, summary.timeToFirstTokenMs

    ```
  </CodeGroup>

  To learn more, read [`GetSummary`](https://pkg.go.dev/github.com/launchdarkly/go-server-sdk-ai/ldai#Tracker.GetSummary) and [`MetricSummary`](https://pkg.go.dev/github.com/launchdarkly/go-server-sdk-ai/ldai#MetricSummary).

  The [Monitoring tab](/docs/home/agentcontrol/monitor) of the config in the LaunchDarkly UI aggregates data from the `track*` functions from across all variations of the config.

  Here's how to record metrics manually:

  <CodeGroup>
    ```ts title="Track duration" lines wrap theme={null}
    // Track your own start and stop time.

    // Set duration to the time (in ms) that your AI model generation takes.
    // The duration may include network latency, depending on how you calculate it.

    aiConfig.tracker.trackDuration(duration);
    ```

    ```ts title="Track token usage" lines wrap theme={null}
    import { LDTokenUsage } from '@launchdarkly/server-sdk-ai';

    // Track your own token usage.

    // First, set up an LDTokenUsage object.
    // Update the input, output, and total fields
    // with return values from your AI model generation.
    const tokens: LDTokenUsage = {
      input: 0,
      output: 0,
      total: 0,
    }

    aiConfig.tracker.trackTokens(tokens);
    ```

    ```ts title="Track output satisfaction rate" lines wrap theme={null}
    import { LDFeedbackKind } from '@launchdarkly/server-sdk-ai';

    // Track your own output satisfaction rate.

    // Pass in LDFeedbackKind.Positive or LDFeedbackKind.Negative.
    aiConfig.tracker.trackFeedback({ kind: LDFeedbackKind.Positive });

    ```

    ```ts title="Track generation (success)" lines wrap theme={null}
    // Track a successful generation event using cfg

    aiConfig.tracker.trackSuccess()
    ```

    ```ts title="Track generation (error)" lines wrap theme={null}
    // Track an unsuccessful generation event using cfg

    aiConfig.tracker.trackError()
    ```

    ```ts title="Track time to first token" lines wrap theme={null}
    // Track the time it takes to generate the first token

    // Pass in the time (in ms) until your first token is generated
    // This may include network latency, depending on how you calculate it

    aiConfig.tracker.trackTimeToFirstToken(1000);
    ```
  </CodeGroup>

  In `completion` mode, the `tracker` is returned from the `completionConfig()` call, so you can access it directly, as in the examples above. Make sure to call `completionConfig()` again each time you use the tracker and generate content from your AI model.

  In `agent` mode, the `tracker` is part of the `agent` returned from the `agentConfig()` or `agentConfigs()` call. If you are working in `agent` mode, replace `tracker` with `agent.tracker` in the examples above.

  The SDK automatically flushes these pending analytics events to LaunchDarkly at regular intervals. If you have a short-lived application, you may need to explicitly request that the underlying LaunchDarkly client deliver any pending analytics events to LaunchDarkly, using [`flush()`](/docs/sdk/features/flush) or [`close()`](/docs/sdk/features/shutdown).

  Here's how:

  <CodeGroup>
    ```js title="Flush tracking events" lines wrap theme={null}
    ldClient.flush();
    ```
  </CodeGroup>

  To learn more, read [`LDAIConfigTracker`](https://launchdarkly.github.io/js-core/packages/sdk/server-ai/docs/interfaces/LDAIConfigTracker.html).
</Accordion>

### Node.js (server-side) AI (turnkey)

<Accordion title="Expand turnkey Node.js (server-side) AI SDK code sample">
  The turnkey Node.js (server-side) AI SDK records AI metrics automatically. You do not obtain a tracker or call `track*` functions. Every `config().invoke()`, `config().stream()`, and `graph().invoke()` call emits generation duration, token usage, generation success or error, and tool-call events on its own. The provider handlers also create OpenTelemetry spans so LaunchDarkly can associate traces with the config.

  Here is an example. Calling the model records metrics automatically:

  <CodeGroup>
    ```ts title="Metrics are recorded automatically" lines wrap theme={null}
    import { config } from '@launchdarkly/ai-server';
    import { createOpenAIHandler } from '@launchdarkly/ai-openai-messages';

    // Calling invoke() records duration, token, generation, and tool-call
    // metrics automatically. No tracker is required.
    const result = await config({
      key: 'example-config-key',
      handler: [createOpenAIHandler()],
    }).invoke('What is feature flagging?', { kind: 'user', key: 'example-user-key' });

    console.log(result.response);
    ```
  </CodeGroup>

  To export traces to LaunchDarkly observability, make the OpenTelemetry SDK packages available at runtime. To learn more, read [Configure observability](/docs/sdk/ai/node-js#configure-observability) in the turnkey Node.js (server-side) AI SDK reference.
</Accordion>

### Python AI

<Accordion title="Expand Python AI SDK code sample">
  If your config uses completion mode, use one of the `track_[model]_metrics` functions to record metrics from your AI model generation. The SDK provides separate `track_[model]_metrics` functions for several of the models that you can select when you set up your [config variations](/docs/home/agentcontrol/create-variation) in the LaunchDarkly user interface.

  If your config uses `agent` mode, you can access the `instructions` returned from the customized config to send to your AI model. Use the `tracker` returned as part of the `agent_config()` or `agent_configs()` functions to record metrics.

  The `tracker` is returned from your call to [customize the config](/docs/sdk/features/agentcontrol-config#python-ai), and is specific to that config. Make sure to call `completion_config` again each time you use the tracker and generate content from your AI model, so that your metrics are correctly associated with the customized config variation.

  Here's how:

  <CodeGroup>
    ```python title="Using OpenAI model, completion mode" expandable lines wrap theme={null}
    if config.enabled:
        # Pass in the result of the OpenAI operation.
        # When calling the OpenAI operation, use details from config.
        # For instance, you can pass config.model.name
        # and config.messages[0].content to your specific OpenAI operation.
        #
        # CAUTION: If the call inside of track_openai_metrics throws an exception,
        # the SDK will re-throw that exception

        messages = [] if config.messages is None else config.messages
        completion = tracker.track_openai_metrics(
            lambda:
              openai_client.chat.completions.create(
                  model=config.model.name,
                  messages=[message.to_dict() for message in messages],
              )
        )
    else:
        # Application path to take when the config is disabled
        pass
    ```

    ```python title="Using Bedrock model, completion mode" lines wrap theme={null}
    if config.enabled:
        # Pass in the result of the Bedrock Converse command.
        # When calling the Bedrock Converse command, use details from config.
        # For instance, you can pass config.model.name
        # and config.messages[0].content to your specific Bedrock Converse command.

        completion = tracker.track_bedrock_converse_metrics(
            client.converse(
                modelId=config.model.name,
                messages=map_messages_to_conversation(config.messages)
            )
        )
    else:
        # Application path to take when the config is disabled
        pass
    ```

    ```python title="Accessing instructions and recording metrics, agent mode" lines wrap theme={null}
    if agent.enabled:
        # Retrieve instructions from the config and pass to your AI model
        result = example_model_api(agent.instructions)

        # Track metrics from the result
        agent.tracker.track_success()
    else:
        # Application path to take when the agent is disabled
        pass
    ```
  </CodeGroup>

  You can use the SDK's other `track*` functions to record metrics manually. This is useful when the SDK does not provide a convenience `track_[model]_metrics` function for your model, or when your application requires streaming, because `track_[model]_metrics` functions expect a response.

  Each `track*` function sends data back to LaunchDarkly. To review the metrics that have been recorded, use `get_summary`:

  <CodeGroup>
    ```python title="Get automatically recorded metrics, completion mode" lines wrap theme={null}
    tracker.get_summary()

    # recorded metrics available in tracker.get_summary().duration, .feedback,
    # .success, .usage, and .time_to_first_token
    ```

    ```python title="Get automatically recorded metrics, agent mode" lines wrap theme={null}
    agent.tracker.get_summary()

    # recorded metrics available in agent.tracker.get_summary().duration, .feedback,
    # .success, .usage, and .time_to_first_token

    ```
  </CodeGroup>

  To learn more, read [`get_summary`](https://launchdarkly-python-sdk-ai.readthedocs.io/en/latest/api-main.html#ldai.tracker.LDAIConfigTracker.get_summary) and [`LDAIMetricSummary`](https://launchdarkly-python-sdk-ai.readthedocs.io/en/latest/api-main.html#ldai.tracker.LDAIMetricSummary).

  The [Monitoring tab](/docs/home/agentcontrol/monitor) of the config in the LaunchDarkly UI aggregates data from the `track*` functions across all variations of the config.

  Here's how to record metrics manually:

  <CodeGroup>
    ```python title="Track duration" maxLines=0 lines wrap theme={null}
    # Track your own start and stop time.

    # Set duration to the time (in ms) that your AI model generation takes.
    # The duration may include network latency, depending on how you calculate it.

    tracker.track_duration(duration)
    ```

    ```python title="Track token usage" lines wrap theme={null}
    # Track your own token usage.

    # TokenUsage is provided by the LaunchDarkly AI SDK.
    # Update the input, output, and total values
    # with return values from your AI model generation.
    tokens = TokenUsage(0, 0, 0)

    tracker.track_tokens(tokens)
    ```

    ```python title="Track output satisfaction rate" lines wrap theme={null}
    # Track your own output satisfaction rate.

    # Pass in FeedbackKind.Positive or FeedbackKind.Negative.
    tracker.track_feedback({"kind": FeedbackKind.Positive})
    ```

    ```python title="Track generation (success)" lines wrap theme={null}
    # Track a successful generation event using your config.

    tracker.track_success()
    ```

    ```python title="Track generation (error)" lines wrap theme={null}
    # Track an unsuccessful generation event using your config.

    tracker.track_error()
    ```

    ```python title="Track time to first token" lines wrap theme={null}
    # Track the time it takes to generate the first token.

    # Pass in the time (in ms) until your first token is generated.
    # This may include network latency, depending on how you calculate it.

    tracker.track_time_to_first_token(1000)
    ```
  </CodeGroup>

  In `completion` mode, the `tracker` is returned from the `completionConfig()` call, so you can access it directly, as in the examples above. Make sure to call `completionConfig()` again each time you use the tracker and generate content from your AI model.

  In `agent` mode, the `tracker` is part of the `agent` returned from the `agentConfig()` or `agentConfigs()` call. If you are working in `agent` mode, replace `tracker` with `agent.tracker` in the examples above.

  The SDK automatically flushes pending analytics events to LaunchDarkly at regular intervals. If you have a short-lived application, you may need to explicitly request that the underlying LaunchDarkly client deliver any pending analytics events to LaunchDarkly, using [`flush()`](/docs/sdk/features/flush) or [`close()`](/docs/sdk/features/shutdown).

  Here's how:

  <CodeGroup>
    ```python title="Flush tracking events" lines wrap theme={null}
    ldclient.get().flush()
    ```
  </CodeGroup>

  To learn more, read [`LDAIConfigTracker`](https://launchdarkly-python-sdk-ai.readthedocs.io/en/latest/api-main.html#ldai.tracker.LDAIConfigTracker).
</Accordion>

### Ruby AI

<Accordion title="Expand Ruby AI SDK code sample">
  Use one of the `track_[model]_metrics` functions to wrap your AI model provider call and record metrics from generation. The SDK provides separate `track_[model]_metrics` functions for several models that you can select when you set up your [config variations](/docs/home/agentcontrol/create-variation) in the LaunchDarkly UI.

  The `tracker` is returned as part of your call to [customize the config](/docs/sdk/features/agentcontrol-config#ruby-ai) and is specific to that config. Make sure to call `config` again each time you use the tracker and generate content from your AI model so your metrics are correctly associated with the customized config variation.

  Here's how to use a provider-specific function to call OpenAI or Bedrock providers and record metrics from your AI model generation:

  <CodeGroup>
    ```ruby title="Using OpenAI model" expandable lines wrap theme={null}
    if ai_config.enabled
      # Wrap the OpenAI operation to record metrics.
      # When calling the OpenAI operation, use details from ai_config.
      # For instance, you can pass ai_config.model.name
      # and ai_config.messages to your specific OpenAI operation.
      #
      # CAUTION: If the call inside track_openai_metrics throws an exception,
      # the SDK will re-throw that exception.

      completion = ai_config.tracker.track_openai_metrics do
        openai_client.chat.completions.create(
          model: ai_config.model.name,
          messages: ai_config.messages.map(&:to_h)
        )
      end
    else
      # Application path to take when the ai_config is disabled
    end
    ```

    ```ruby title="Using Bedrock model" expandable lines wrap theme={null}
    if ai_config.enabled
      # Wrap the Bedrock Converse command to record metrics.
      # When calling the Bedrock Converse command, use details from ai_config.
      # For instance, you can pass ai_config.model.name
      # and ai_config.messages to your specific Bedrock Converse command.

      completion = ai_config.tracker.track_bedrock_converse_metrics do
        bedrock_client.converse(
          map_converse_arguments(
            ai_config.model.name,
            ai_config.messages
          )
        )
      end
    else
      # Application path to take when the ai_config is disabled
    end
    ```

    ```ruby title="Bedrock helper function" maxLines=0 expandable lines wrap theme={null}
    # The map_converse_arguments helper function transforms the messages
    # to fit the Bedrock SDK.

    def map_converse_arguments(model_id, messages)
      args = {
        model_id: model_id,
      }

      mapped_messages = []
      user_messages = messages.select { |msg| msg.role == 'user' }
      mapped_messages << { role: 'user', content: user_messages.map { |msg| { text: msg.content } } } unless user_messages.empty?

      assistant_messages = messages.select { |msg| msg.role == 'assistant' }
      mapped_messages << { role: 'assistant', content: assistant_messages.map { |msg| { text: msg.content } } } unless assistant_messages.empty?
      args[:messages] = mapped_messages unless mapped_messages.empty?

      system_messages = messages.select { |msg| msg.role == 'system' }
      args[:system] = system_messages.map { |msg| { text: msg.content } } unless system_messages.empty?

      args
    end
    ```
  </CodeGroup>

  You can use the SDK's other `track*` functions to record these metrics manually. You may need to do this if you are using a model for which the SDK does not provide a convenience `track_[model]_metrics` function. The `track_[model]_metrics` functions are expecting a response, so you may also need to do this if your application requires streaming.

  Each of the `track*` functions sends data back to LaunchDarkly. To review the metrics that have been recorded, use the `summary` property in the `tracker`:

  <CodeGroup>
    ```ruby title="Get automatically recorded metrics" lines wrap theme={null}
    ai_config.tracker.summary

    # recorded metrics available in tracker.summary include: duration, feedback,
    # success, usage, and time_to_first_token

    ```
  </CodeGroup>

  To learn more, read [`summary`](https://launchdarkly.github.io/ruby-server-sdk-ai/LaunchDarkly/Server/AI/AIConfigTracker.html#summary-instance_method) and [`MetricSummary`](https://launchdarkly.github.io/ruby-server-sdk-ai/LaunchDarkly/Server/AI/MetricSummary.html).

  The [Monitoring tab](/docs/home/agentcontrol/monitor) of the config in the LaunchDarkly UI aggregates data from the `track*` functions from across all variations of the config.

  Here's how to record metrics manually:

  <CodeGroup>
    ```ruby title="Track duration" lines wrap theme={null}
    # Track your own start and stop time.

    # Set duration to the time (in ms) that your AI model generation takes.
    # The duration may include network latency, depending on how you calculate it.

    ai_config.tracker.track_duration(duration)
    ```

    ```ruby title="Track token usage" lines wrap theme={null}
    # Track your own token usage.

    # First, create an instance of TokenUsage.
    # Update the input, output, and total values
    # with return values from your AI model generation.
    tokens = LaunchDarkly::Server::AI::TokenUsage.new(total: 300, input: 200, output: 100)

    ai_config.tracker.track_tokens(tokens)
    ```

    ```ruby title="Track output satisfaction rate" lines wrap theme={null}
    # Track your own output satisfaction rate

    # Pass in kind: :positive or kind: :negative
    ai_config.tracker.track_feedback(kind: :positive)
    ```

    ```ruby title="Track generation (success)" lines wrap theme={null}
    # Track each time there was a successful generation event using your config

    ai_config.tracker.track_success
    ```

    ```ruby title="Track generation (error)" lines wrap theme={null}
    # Track each time there was an unsuccessful generation event using your config

    ai_config.tracker.track_error
    ```

    ```ruby title="Track time to first token" lines wrap theme={null}
    # Track the time it takes to generate the first token

    # Pass in the time (in ms) until your first token is generated
    # This may include network latency, depending on how you calculate it

    ai_config.tracker.track_time_to_first_token(1000)
    ```
  </CodeGroup>

  Make sure to call `config` again each time you use the tracker and generate content from your AI model.

  The SDK automatically flushes these pending analytics events to LaunchDarkly at regular intervals. If you have a short-lived application, you may need to explicitly request that the underlying LaunchDarkly client deliver any pending analytics events to LaunchDarkly, using [`flush`](/docs/sdk/features/flush) or [`close`](/docs/sdk/features/shutdown).

  Here's how:

  <CodeGroup>
    ```ruby title="Flush tracking events" lines wrap theme={null}
    ld_client.flush()
    ```
  </CodeGroup>

  To learn more, read [`ConfigTracker`](https://launchdarkly.github.io/ruby-server-sdk-ai/LaunchDarkly/Server/AI/AIConfigTracker.html).
</Accordion>