Online evaluations in AI Configs

Overview

Online evaluations let you measure the quality of AI Config variations in production by attaching built-in judges that score model responses for accuracy, relevance, and toxicity.

A judge is a specialized AI Config that uses an evaluation prompt to score responses from another AI Config and return a numeric score. LaunchDarkly provides three built-in judges:

• Accuracy
• Relevance
• Toxicity

You can attach judges to completion-mode AI Config variations in the LaunchDarkly UI. For other variations, invoke a judge programmatically using the AI SDK.

The sections below focus on using these built-in judges to monitor and evaluate model quality in production. You can extend the same evaluation framework with custom judges if you need to measure additional or domain-specific quality signals.

When attached to an AI Config variation, built-in judges run automatically and record evaluation scores as metrics.

Evaluation scores appear on the Monitoring tab for each variation, along with latency, cost, and user satisfaction metrics. These scores provide a continuous view of model behavior in production and help you detect regressions or understand how changes to prompts or models affect performance.

Online evaluations differ from offline or pre-deployment testing. Offline evaluations run against test datasets or static examples. Online evaluations run as your application sends real traffic through an AI Config.

Online evaluations work alongside observability. Observability shows model responses and routing details. Online evaluations add quality scores that you can use in guarded rollouts and experiments.

Use online evaluations to:

• Monitor model behavior during production use
• Detect changes in quality after a rollout
• Trigger alerts or rollback actions based on evaluation scores
• Compare variations using live performance data

How online evals work

Online evaluations add automated quality checks to AI Configs. Each evaluation produces a score between 0.0 and 1.0. Higher scores indicate better results from the attached judge.

When a variation generates a model response, LaunchDarkly runs the attached judge in the background. The judge uses its evaluation prompt to score the response.

1. The primary AI Config generates a model response.
2. The judge evaluates the response using its evaluation prompt.
3. The judge returns structured results that include numeric scores and brief explanations such as "score": 0.9, "reason": "Accurate and relevant answer".
4. LaunchDarkly records these results as metrics and displays them on the Monitoring tab.

Evaluations run asynchronously and respect your configured sampling rate. You can adjust sampling to balance cost and visibility.

The following example shows a conceptual summary of how multiple built-in judges might score a single response across different evaluation dimensions. Each judge returns its own score and reasoning.

{
  "accuracy": { "score": 0.85, "reasoning": "Answered correctly with one minor omission" },
  "relevance": { "score": 0.92, "reasoning": "Directly addresses the user request" },
  "toxicity": { "score": 1.00, "reasoning": "No harmful or unsafe phrasing detected" }
}

LaunchDarkly records evaluation results as metrics that are consistent across providers and environments.

Run judge evaluations programmatically

In addition to attaching judges in the UI, you can evaluate arbitrary input and output pairs directly using the AI SDK and a judge key.

This approach:

• Does not require attaching a judge to a completion-mode variation
• Can be used to evaluate outputs from agent-based workflows
• Lets you evaluate responses from custom pipelines or external systems

The following Python example shows how to evaluate input and output directly using a judge:

1
import asyncio
2
import ldclient
3
from ldclient.config import Config
4
from ldclient import Context
5
from ldai import LDAIClient, AICompletionConfigDefault
6
7
async def main():
8
    # Initialize LaunchDarkly
9
    ldclient.set_config(Config("YOUR_SDK_KEY"))
10
    ai_client = LDAIClient(ldclient.get())
11
12
    # Create evaluation context
13
    context = Context.builder("example-user-key").kind("user").build()
14
15
    # Fallback if judge is unavailable
16
    fallback = AICompletionConfigDefault(enabled=False)
17
18
    # Retrieve judge configuration
19
    judge = await ai_client.create_judge(
20
        "your-judge-key",
21
        context,
22
        fallback
23
    )
24
25
    if judge and judge.enabled:
26
        input_text = "User question"
27
        output_text = "Model response"
28
29
        result = await judge.evaluate(input_text, output_text)
30
31
        if result:
32
            print(result.to_dict())
33
34
asyncio.run(main())

For a complete, production-ready example including initialization checks and error handling, see the Python direct_judge_example.py on GitHub.

In this example:

• create_judge() retrieves the judge configuration for the provided context.
• evaluate() scores the input and output pair.
• The returned result includes structured evaluation data such as scores and reasoning.

If you want to record evaluation scores as metrics associated with an AI Config, explicitly track the returned evaluation scores in your application code using the AI Config’s tracker. Programmatic judge evaluation does not automatically emit monitoring metrics.

Programmatic judge evaluation does not attach judges to variations in the UI and does not automatically enable Monitoring tab metrics, guarded rollout integration, or experiment metric selection.

Set up and manage judges

Attach judges to variations

To attach a judge to a variation in the LaunchDarkly UI, the AI Config must use completion mode. Attach the judge as follows:

1. In LaunchDarkly, click AI Configs.

2. Click the name of the Completion AI Config you want to edit.

3. Select the Variations tab.

4. Open a variation or create a new variation.

5. In the “Judges” section, click + Attach judges.

   

6. Select one or more of the judges. If necessary, enter text in the search bar to search for a judge. By default, LaunchDarkly provides three judges: Accuracy, Relevance, and Toxicity.

7. (Optional) Set the sampling percentage to control how many model responses are evaluated.

8. Click Review and save.

Attached judges remain connected to the variation until you remove them.

Adjust sampling or detach judges

You can adjust sampling or detach judges at any time from the “Judges” section of a variation.

From this section, you can:

• Raise or lower the sampling percentage
• Disable a judge by setting its sampling percentage to 0 percent
• Remove a judge by clicking its X icon

Run the SDK example

You can use the LaunchDarkly Node.js (server-side) AI SDK example to confirm that online evaluations run as expected. The SDK evaluates chat responses using attached judges.

To set up the SDK example:

1. Clone the LaunchDarkly JavaScript SDK repository.
2. From the repository root, follow the setup instructions in the judge evaluation example README.
3. Configure your environment with your LaunchDarkly project key, environment key, and model provider credentials.
4. Start the example.

The example shows how to evaluate responses using attached judges. Judges run asynchronously and do not block application responses. Evaluation metrics appear on the Monitoring tab within one to two minutes.

View results from the Monitoring tab

Open the Monitoring tab for your AI Config to view evaluation results.

When you attach one or more of the built-in judges, LaunchDarkly records an evaluation metric for each judge you attach:

For each attached judge, the Monitoring tab displays charts with recent and average scores over the selected time range. You can view individual results and reasoning details for each data point. Metrics update as evaluations run.

These metrics appear both on the Monitoring tab and in the Metrics list for your project.

Use evaluation metrics in guardrails and experiments

Evaluation metrics appear as selectable metrics in guarded rollouts and experiments for AI Configs in completion mode.

• In guarded rollouts, you can pause or revert a rollout when evaluation scores fall below a threshold.
• In experiments, you can use evaluation metrics as experiment goals to compare variations.

This creates a connected workflow for releasing and evaluating changes to prompts and models.

Privacy and data handling

Online evaluations run within your LaunchDarkly environment using your configured model providers. LaunchDarkly does not store or share your prompts, model responses, or evaluation data with any third-party systems.

For more information, read AI Configs and information privacy.