Feature flags and AI ConfigsAI ConfigsOperate AI ConfigsDeliver and monitor AI Configs

Monitor AI Configs

Overview

This topic explains how to monitor the performance of your AI Configs. Performance metrics for AI Configs are available in the LaunchDarkly user interface if you track AI metrics in your SDK.

Monitoring metrics collected from your SDK apply to both completion mode and agent mode. You can attach judges to completion-mode AI Config variations in the LaunchDarkly UI. For other variations, invoke a judge programmatically using the AI SDK. To learn more, read Online evaluations in AI Configs.

Metrics and traces are separate

The Monitoring tab displays metrics that your application records using a LaunchDarkly AI SDK, such as latency, token usage, cost, and evaluation scores. These metrics are separate from traces.

When you configure the LaunchDarkly Observability plugin with the AI SDK and spans are successfully ingested, LaunchDarkly associates traces with the evaluated AI Config. You can then view related traces from the AI Config detail page.

Collect data for AI Config variations

Data appears on the Monitoring tab for an AI Config when you record metrics from your AI model generation. For each AI SDK, the function to record metrics takes a completion from your AI model generation, so you can make the call to your AI model provider and record metrics from model generation in one step. You can record duration, token usage, generation success and error, time to first token, output satisfaction, and more.

To learn how, read Tracking AI metrics. For a detailed example, read Step 4 in the Quickstart for AI Configs.

You can also monitor variation rollouts

You can use a guarded rollout when you are releasing new variations to your customers. The Monitoring tab shows a guarded rollout’s progress and SDK-recorded metric results. LaunchDarkly highlights regressions and pauses a guarded rollout automatically if needed.

Monitor an AI Config

To monitor the performance of an AI Config:

  1. Navigate to the detail page for the AI Config and select the Monitoring tab.
  2. At the top of the tab, use the controls to specify the monitoring data you want to view:
    • Use the Environment dropdown to select the environment you want to monitor. Performance metrics are specific to each environment.
    • Select a date range.
    • Use the Charts dropdown to select which set of charts to view.
  3. Review charts of the available monitoring data. The data in each chart applies to the selected environment and is broken out by variation:
    • The Tokens charts show average input and output tokens used by the AI Config, by variation.
    • The Satisfaction chart shows the percentage of “thumbs up” ratings provided by end users who have encountered the AI Config in this environment, by variation.
    • The Generations chart shows the average number of successful generations completed using the AI Config in this environment, by variation.
    • The Time to generate chart shows the average duration per generation. This represents the total duration of calls to your LLM provider divided by the number of total generations, by variation.
    • The Error rate chart shows the percentage of errors out of the total number of generations attempted, by variation.
    • The Time to first token chart shows the mean time it takes to generate the initial token.
    • Costs chart shows the sum of the input token cost and output tokens cost used by the AI Config, by variation.
  4. Review the table of available monitoring data. The table includes the data displayed in the charts as well as additional data, such as both average and total input and output tokens.
  5. (Optional) In the table, select the variations or versions for which you want to view data. The charts update based on your selection.
  6. (Optional) Click Export data as CSV to download a CSV file for further analysis.

For data to appear on the Monitoring tab, you must record the metrics in your application using a track call from a LaunchDarkly AI SDK. To learn more, read Tracking AI metrics.

The data on the Monitoring tab updates approximately every minute. For each metric, results are broken out by AI Config variation. Metrics with no results display “No data.” If no data exists for a particular variation, that variation is not included in the total shown at the top of the metric card.

Here is a partial image of the Monitoring tab:

The Monitoring tab for an AI Config.

The "Monitoring" tab for an AI Config.

The Monitoring tab displays performance metrics for a single AI Config in a selected environment. It supports detailed analysis of variation performance, guarded rollout progress, and version-level comparisons within that AI Config.

For project-level monitoring across multiple AI Configs, use AI insights.

The Trends explorer tab aggregates performance data across multiple AI Configs, variations, models, providers, and targeting rules. It lets you compare metrics over time and identify trends across configurations.

Trends explorer includes:

  • Time series charts across AI Configs, variations, models, providers, and targeting rules
  • Filtering to focus on specific variations, models, or rules
  • Version change annotations to correlate updates with performance trends
  • Summary metrics for cost, generations, token usage, and satisfaction
  • CSV export of aggregated performance data

Use the following guidance to choose the appropriate view:

  • Use Monitoring to diagnose and optimize a specific AI Config and its variations
  • Use AI insights to monitor cost, usage, and quality across multiple AI Configs at the project level
  • Use Trends explorer to compare metrics across AI Configs, models, and providers over time

To learn more about trends explorer features, read View AI Config trends explorer.