Skip to main content
This topic explains how to run AI evaluations from your own code or CI workflow and send the results to LaunchDarkly for inspection and comparison. Running evaluations from code lets you run your own application code against a dataset. The Python and Node.js AI SDKs can run the judges and scorers you choose against each generated output and send the scores to LaunchDarkly. LaunchDarkly records the results for inspection and comparison. Running evaluations from code lets you:
  • Run evaluations from a script or a CI workflow, using the same code and model provider your application uses.
  • Score outputs with deterministic checks that you write in code, with LaunchDarkly judges that use a model to grade a response, or with both in the same run.
  • Evaluate the tool calls an agent makes to reach its answer, such as which tools it calls, in what order, and with what arguments.
  • Define datasets and tools directly in your code, or manage them in the LaunchDarkly Library.
  • Review run results in LaunchDarkly and compare them over time.
  • Use the pass or fail result from a run to gate a release in CI.
Using the same dataset and criteria across runs ensures consistent comparisons. This helps you identify meaningful improvements and avoid introducing regressions.

​
How evaluations from code work

You start an evaluation, which is a test of your handler against a dataset, with a single call to run() in the AI SDK. Each run() call creates one evaluation with one run. The call has two phases:
  1. Generation: The SDK gets the dataset rows, either from a dataset in LaunchDarkly or from rows you pass in code. It calls your handler one time for each row. The handler calls the model and returns the output. You can use one of the LaunchDarkly provider handlers or write a custom handler. To learn more, read Choose a provider handler.
  2. Evaluation: The SDK scores each generated output with the criteria you pass. A criterion is a check applied to each output, and it is either a judge or a scorer.
The SDK sends every result to LaunchDarkly as an event. LaunchDarkly derives each row verdict from the reported scores and configured thresholds. When the run completes, the SDK polls the run summary and returns the run URL, the summary, and a passed value. To fail a CI build when a run fails, set your script’s exit code from passed. In the Python SDK, passed is false if any row failed, returned an error, or is still pending. To learn more, read How verdicts work. To run evaluations from code, you need an API access token and an SDK key. In LaunchDarkly, the API access token is called the Eval API key. To start generating your credentials:
  1. Click Evaluations in the side navigation.
  2. Click Run evaluations.
  3. Choose From code. The “Run evals from code” dialog appears.
To learn how to complete the dialog and copy your credentials, read Get your credentials. To learn how to write the script, read Evaluations from code in the AI SDKs. To learn about the limits that apply to runs, read Limits.

​
Judges and scorers

Scorers and judges grade outputs in different ways:
  • Scorers run locally in your script. Use a scorer for deterministic checks, such as whether the output contains an expected value. A scorer receives the dataset row and the generated output, and returns a number from 0 to 1. A yes-or-no check returns 1.0 or 0.0.
  • Judges run through a model. Use a judge for qualities that are hard to check with code, such as accuracy or tone. The judge must exist in your LaunchDarkly project before a run can use it, and your script references it by key. To learn how to create a judge, read Judges.
You can use judges, scorers, or both in the same run. Because a judge is a model call, it requires a provider. A judge runs through your handler when that handler serves the judge’s provider. You can use a different model provider for your judges than you use for generation. For example, you can generate with OpenAI and grade with Anthropic. When the judge uses a different provider, you pass handlers for both providers in handlers, and the SDK selects the handler that matches each provider. To learn how, read Score rows in Python or Score rows in Node.js.

​
How verdicts work

Each criterion has a threshold, which is the score a row must meet to pass that criterion. LaunchDarkly compares each row’s score with the threshold to decide the row verdict, which is whether that row passed. The run verdict is whether the whole run passed. By default, a scorer’s threshold is 1.0, which means a row passes only on a perfect score. A judge’s default threshold is 0.5. The defaults are the same in both SDKs. The run fails when any row fails, and the run returns an error when any row returns an error. Otherwise, the run passes. If your handler raises an error for a row, LaunchDarkly records that row as ERROR and the other rows continue. The SDK does not retry the handler for that row.

​
Tool trajectory

Tools are optional. If your agent calls tools, the SDK records each call, and a judge can grade how the agent used them. A tool trajectory is the record of the tool calls an agent made while producing a response, including the order, the arguments, and each result or error. If your agent calls tools, a judge can grade the trajectory along with the answer itself. The SDK records every tool call a row makes and gives the record to the judge. A judge rubric can then ask whether the agent called the right tool, called tools in the right order, used the right arguments, and recovered from a failed call. Keep the following in mind:
  • Scorers cannot see the trajectory. To assert something about tool use, use a judge.
  • Tools that your agent calls must be defined, either in the LaunchDarkly Library or inline in your code. To learn more, read Use Library datasets and tools and Define datasets and tools in code.
  • Tools that run inside a model provider, called provider built-in tools, do not appear in the trajectory.
To learn about the limits that apply to tools, read Limits.

​
Use Library datasets and tools

You can run an evaluation with datasets and tools that are stored in the Library. Library assets are managed in LaunchDarkly, and you reference them from your code by key.

​
Library datasets

A Library dataset is a CSV or JSONL file that you upload to LaunchDarkly. To run an evaluation against one, pass its dataset key to run() instead of passing rows. Use the dataset’s key, not its ID. The dataset must exist in your project before the run starts. To learn how to upload a dataset, read Datasets.

​
Library tools

A Library tool is a tool that you create once in LaunchDarkly and reuse. The Library holds the tool’s key, description, and schema. To use one in a run, reference it by its tool key and provide the implementation in your code. LaunchDarkly supplies the description and schema that the model sees, and your code runs the tool. If a tool key is not in the Library and was not defined inline, the run fails before it creates any records. To learn how to create a tool, read Tools.

​
Define datasets and tools in code

You can also define your datasets and tools directly in your code. Your script brings these assets with it, and LaunchDarkly stores and displays the results of your runs in the same way.

​
Inline datasets

An inline dataset is a test set that you define in your code. Instead of passing a dataset key, you pass the rows directly. For example, your script can load sample data from a file and pass the values to run(). When you use an inline dataset:
  • LaunchDarkly records the run and shows the same results as for a Library dataset.
  • LaunchDarkly marks the run as inline, which means you can tell that the dataset was defined in code and not hosted in the Library.
Each inline dataset row can include the following fields:
  • input: The prompt or request. This is a template, and {{name}} placeholders resolve from variables.
  • expectedOutput: Optional. The target output. Scorers and judges can read it.
  • variables: Optional. Named values for the placeholders in input.
  • metadata: Optional. A JSON object that is stored with the row.
  • rowIdx: Optional. The row’s zero-based position. If provided, it must match the row’s position in the array.
These are the field names for rows passed directly to run(). The DatasetRow object received by your handler or scorer uses SDK-specific names: Python uses row_index and expected_output, while Node.js uses rowIndex and expectedOutput.

​
Inline tools

You only need to define tools if your agent calls them, and they matter most when you want a judge to grade the tool trajectory. An inline tool is a tool that you define in code. You provide the following:
  • Key: The identifier for the tool, such as lookup_order.
  • Implementation function: The function that runs when the model calls the tool.
  • Schema: A JSON schema that tells the model which arguments to provide.
  • Description: A description that tells the model what the tool does. The model uses the description to decide when to call the tool.
LaunchDarkly marks inline tools as inline in the evaluation results. A typical workflow is to define tools in code during early development, and then define them in the Library for production.

​
Review evaluation runs

When a run completes, the script prints the run ID, a URL, the summary, and whether the run passed. Open the URL to review the run in LaunchDarkly. To review example output for each SDK, read Run your first evaluation in Python or Run your first evaluation in Node.js. You can also find a run from the LaunchDarkly side navigation. To review the results of an evaluation that you ran from code:
  1. Click Evaluations in the side navigation. The “Evaluations” page appears.
  2. Click the From code tab to display all evaluations that you ran from code.
  3. Click the evaluation row you want more information about. The “Results” page appears.
  4. Review the information associated with the run, including the run summary, the verdict, and the dataset.
The “Results” page includes the following information:
  • Evaluation details: The model, the system prompt and user prompt, the tools and their versions, the dataset, and who ran the evaluation and when.
  • Run summary: The verdict, such as “Passed,” and the number of rows evaluated. It also shows the dataset file name and row count, and when the run started and ended.
  • Row counts: The percentage of rows processed, and how many rows passed, failed, or remain.
  • Run metrics: The total run cost, average tokens per row, and P50 and P95 latency.
  • Criteria results: For each criterion, the pass rate, the number of rows that failed, the P50 and P95 scores, and the worst score.
  • Row results: For each row, the input, variables, expected output, and generated output, along with the score and reasoning from each criterion. Each row also shows its average latency and tokens, and a summary of its tool calls. Click Trace to see more detail about a row, or click Failed only to show only the rows that failed.