> ## Documentation Index
> Fetch the complete documentation index at: https://launchdarkly.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations from code

This topic explains how to run evaluations from code with the LaunchDarkly [Python AI SDK](https://launchdarkly.com/docs/sdk/ai/python) or [Node.js AI SDK](https://launchdarkly.com/docs/sdk/ai/node-js).

In addition to running evaluations in the playground, you can run them from your own code. This lets you use your own generation code against a dataset, score each output, and send the results to LaunchDarkly. LaunchDarkly stores the results. With the Python and Node.js AI SDKs, you can:

* Generate an output for every row of a dataset, using a handler for your model provider or a handler that you write.
* Score each output with deterministic scorers that you write in code, with LaunchDarkly judges, or with both.
* Grade the tool calls an agent makes to reach its answer, such as which tools it calls, in what order, and with what arguments.
* Define datasets and tools inline in your code, or use datasets and tools from the LaunchDarkly Library.
* Use the pass or fail result of a run to set your script's exit code, for example in a CI workflow.

To learn more about the concepts behind this feature, read [Evaluations from code](https://launchdarkly.com/docs/home/agentcontrol/evaluations-from-code.md).

## How evaluations from code work

A single call to `run()` generates an output for each dataset row and scores it. Then, it returns the run URL, the summary, and whether the run passed. LaunchDarkly derives each row verdict from the scores, and the SDK reads the run summary to report whether the run passed. To learn more, read [How evaluations from code work](https://launchdarkly.com/docs/home/agentcontrol/evaluations-from-code.md#how-evaluations-from-code-work).

Keep the following in mind as you write code to run evaluations:

* The SDK needs an API access token and an SDK key.
  * API access token: Resolves datasets, tools, and judges, creates the evaluation and the run, and polls the summary.
  * SDK key: Sends every generation and criterion result to LaunchDarkly as an event. Results reach LaunchDarkly only through this channel.
* Every call to `run()` creates a new evaluation. Use a new `key` for each new run, such as one that includes a timestamp. To retry a failed run, reuse its key.
* If your handler raises an error for a row, LaunchDarkly records the row as `ERROR` and the other rows continue.
* The SDK does not retry a handler for a row that raises an error.

## Prerequisites

To run your first evaluation, you need:

* A LaunchDarkly project.
* One of the following AI SDKs:
  * The [Python AI SDK](https://launchdarkly.com/docs/sdk/ai/python) `launchdarkly-ai-server` 1.0.0 or later, with Python 3.12 or later.
  * The [Node.js (server-side) AI SDK](https://launchdarkly.com/docs/sdk/ai/node-js) `@launchdarkly/ai-node` 1.0.0 or later, with Node.js 22 or later.
* An API access token and the environment's server-side SDK key. To learn how to get them, read [Get your credentials](#get-your-credentials).
* One handler package for the model provider you generate with, and the API key or credentials for that provider. You do not need a provider key to verify your setup. To learn how, read [Run your first evaluation in Python](#run-your-first-evaluation-in-python) or [Run your first evaluation in Node.js](#run-your-first-evaluation-in-node-js).
* A dataset, either in the Library or defined inline in your code. To learn more, read [Datasets](https://launchdarkly.com/docs/home/agentcontrol/datasets.md).

Depending on how you score rows and what your agent does, you might also need:

* A judge, if you score with LaunchDarkly judges. The judge must exist in the project as a judge config. To learn more, read [Judges](https://launchdarkly.com/docs/home/agentcontrol/judges.md).
* A tool definition for each tool your agent calls, in the Library or inline in your code. To learn more, read [Tools](https://launchdarkly.com/docs/home/agentcontrol/tools.md).

## Get your credentials

The SDK needs two credentials: an API access token and an SDK key. In LaunchDarkly, the API access token is called the **Eval API key**. To learn what each credential does, read [How evaluations from code work](#how-evaluations-from-code-work).

> [!WARNING]
>
> The SDK key is required. Without it, no result is ingested and every run times out. `init_evaluations()` in Python and `initEvaluations()` in Node.js raise an error immediately if the key is missing.

### Generate your credentials

To generate your credentials in LaunchDarkly:

1. Click **Evaluations** in the side navigation.
2. Click **Run evaluations**, then choose **From code**. The "Run evals from code" dialog appears.
3. Choose how you want to set up:
   * **Set up with an agent** gives you instructions to hand to your agent.
   * **Set up manually** gives you example code for the language and provider you select. The "Connect your code" dialog appears.
4. (Optional) If you plan to use the credentials in CI or other automation, check **Create a service token** under **Eval API key**.
5. Click **Generate keys**. The "Finish setup" dialog appears with your new credentials and a `.env` file.
6. Copy the credentials, the `.env` file, and the information specific to your choice:
   * **Set up with an agent:** The instructions for your agent.
   * **Set up manually:** Choose your programming language and preferred provider to generate an install command and a code example.
7. Click **Done**.

You can also create an API access token and copy an SDK key yourself. To learn how, read [API access tokens](https://launchdarkly.com/docs/home/account/api-create.md) and [SDK keys](https://launchdarkly.com/docs/home/account/environment/keys.md). If you create the token yourself, assign a role that allows reading datasets and tools, and creating evaluations and evaluation runs, in the target project. The **Writer** role works.

### Write the .env file

If you generated your credentials in LaunchDarkly, the "Finish setup" dialog gives you a `.env` file with your credentials that you can download. It does not include your project key or dataset key. Add `LD_PROJECT_KEY` and `LD_DATASET_KEY` to it.

Otherwise, create a `.env` file. Keep only secrets and project identifiers in your `.env` file. Model names, tool keys, judge keys, and thresholds live in your scripts. For security, add the `.env` to your `.gitignore` file.

Here is an example:

```text title=".env file" lines wrap theme={null}
LD_API_TOKEN=api-...
LD_SDK_KEY=sdk-...
LD_PROJECT_KEY=your-project-key
LD_DATASET_KEY=your-dataset-key

# Only for runs that call OpenAI
OPENAI_API_KEY=sk-...
```

### Add your provider credentials

LaunchDarkly credentials and model provider credentials authorize different things. Your LaunchDarkly credentials let the SDK create the evaluation, report results, and read the run summary. Your provider credentials let your handler call the model. Add your provider's API key to your `.env` file, such as `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`. If your provider uses other credentials, such as Amazon Bedrock through a LangChain handler or a custom handler, configure them the way that provider's SDK expects.

To learn more, read [Choose a provider handler](#choose-a-provider-handler). To write your own handler, read [Use a custom handler](#use-a-custom-handler).

## Choose a provider handler

The `handler` argument to `run()` accepts any LaunchDarkly handler. Each handler reads the provider, model, mode, parameters, and instructions or messages from the `generation` argument. Set `generation.provider` and `generation.mode` to match the handler.

The handler packages for each SDK are:

**Python**

<table>
  <thead>
    <tr>
      <th>Package</th>
      <th>Factory</th>
      <th>Use when</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`launchdarkly-ai-openai-messages`</td>
      <td>`create_openai_messages_handler()`</td>
      <td>OpenAI, single-turn or tool-calling chat</td>
    </tr>

    <tr>
      <td>`launchdarkly-ai-openai-agents`</td>
      <td>`create_openai_agent_handler()`</td>
      <td>OpenAI Agents SDK</td>
    </tr>

    <tr>
      <td>`launchdarkly-ai-claude-messages`</td>
      <td>`create_claude_messages_handler()`</td>
      <td>Anthropic Messages API</td>
    </tr>

    <tr>
      <td>`launchdarkly-ai-claude-agents`</td>
      <td>`create_claude_agents_handler()`</td>
      <td>Claude Agent SDK</td>
    </tr>

    <tr>
      <td>`launchdarkly-ai-langchain-messages`</td>
      <td>`create_langchain_messages_handler()`</td>
      <td>Any LangChain chat model</td>
    </tr>

    <tr>
      <td>`launchdarkly-ai-langchain-agents`</td>
      <td>`create_langchain_agents_handler()`</td>
      <td>LangChain agents</td>
    </tr>

    <tr>
      <td>Your own function</td>
      <td>`create_handler()`</td>
      <td>A provider that the SDK has no package for, an internal gateway, or replaying stored outputs. To learn more, read [Use a custom handler](#use-a-custom-handler).</td>
    </tr>
  </tbody>
</table>

**Node.js**

<table>
  <thead>
    <tr>
      <th>Package</th>
      <th>Factory</th>
      <th>Use when</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`@launchdarkly/ai-openai-messages`</td>
      <td>`createOpenAIHandler()`</td>
      <td>OpenAI, single-turn or tool-calling chat</td>
    </tr>

    <tr>
      <td>`@launchdarkly/ai-openai-agents`</td>
      <td>`createOpenAIAgentHandler()`</td>
      <td>OpenAI Agents SDK</td>
    </tr>

    <tr>
      <td>`@launchdarkly/ai-claude-messages`</td>
      <td>`createClaudeMessagesHandler()`</td>
      <td>Anthropic Messages API</td>
    </tr>

    <tr>
      <td>`@launchdarkly/ai-claude-agents`</td>
      <td>`createClaudeAgentsHandler()`</td>
      <td>Claude Agent SDK</td>
    </tr>

    <tr>
      <td>`@launchdarkly/ai-langchain-messages`</td>
      <td>`createLangChainHandler()`</td>
      <td>Any LangChain chat model</td>
    </tr>

    <tr>
      <td>`@launchdarkly/ai-langchain-agents`</td>
      <td>`createLangChainAgentsHandler()`</td>
      <td>LangChain agents</td>
    </tr>

    <tr>
      <td>Your own function</td>
      <td>`createHandler()`</td>
      <td>A provider that the SDK has no package for, an internal gateway, or replaying stored outputs. To learn more, read [Use a custom handler](#use-a-custom-handler).</td>
    </tr>
  </tbody>
</table>

To use a different provider, install its handler package, pass its factory as `handler`, and set `generation.provider` and `generation.mode` to match. For example, to generate with Anthropic:

```python title="Python AI SDK" lines wrap theme={null}
from launchdarkly_ai_claude_messages import create_claude_messages_handler

result = await evals.run(
    key=f"first-eval-anthropic-{unique}",
    dataset=dataset_key,
    handler=create_claude_messages_handler(),
    generation={
        "provider": "Anthropic",
        "model": "claude-sonnet-4-5",
        "mode": "completion",
        "instructions": "Answer the user's question accurately and concisely.",
    },
)
```

```typescript title="Node.js (server-side) AI SDK" lines wrap theme={null}
import { createClaudeMessagesHandler } from '@launchdarkly/ai-claude-messages';

const result = await evals.run({
  key: `first-eval-anthropic-${unique}`,
  dataset: datasetKey,
  handler: createClaudeMessagesHandler(),
  generation: {
    provider: 'Anthropic',
    model: 'claude-sonnet-4-5',
    mode: 'completion',
    messages: [
      {
        role: 'system',
        content: "Answer the user's question accurately and concisely.",
      },
    ],
  },
});
```

Set `ANTHROPIC_API_KEY` in your `.env` file for this handler.

If no package fits, write a custom handler. To learn how, read [Use a custom handler](#use-a-custom-handler).

### Use a custom handler

A custom handler is an async function that calls your model and returns the output. Wrap it with `create_handler()` in Python or `createHandler()` in Node.js to declare the provider and mode it supports. The SDK calls it once for each row. Use a custom handler to call a provider the SDK has no package for, to call an internal gateway, or to replay stored outputs.

Here is an example:

```python title="Python AI SDK" lines wrap theme={null}
from typing import Any

from launchdarkly_ai_server import create_handler


async def handler(
    config: dict[str, Any],
    user_input: str | None = None,
    tool_handlers: dict[str, Any] | None = None,
    variables: dict[str, Any] | None = None,
) -> dict[str, Any]:
    return {"output": f"response for {user_input or ''}"}


handler = create_handler(("custom", "completion"), handler)
```

```typescript title="Node.js (server-side) AI SDK" lines wrap theme={null}
import { createHandler } from '@launchdarkly/ai-node';

const handler = createHandler(
  ['custom', 'messages'],
  async (
    config,
    userInput,
    toolHandlers,
    variables,
    history,
  ) => {
    return {
      output: `response for ${userInput ?? ''}`,
    };
  },
);
```

The first argument to `create_handler()` or `createHandler()` sets the provider and mode that the handler serves. The handler receives the following arguments:

**Python**

* `config` holds the provider, model, parameters, instructions or messages, and the tool definitions from `generation`.
* `user_input` is the rendered row input.
* `tool_handlers` maps each tool key to the function you passed in `tools`.
* `variables` holds the row variables plus the rendered `input` and `expected_output`.

Return a dict with `output`. Add `usage` with `input_tokens` and `output_tokens` when you have them.

**Node.js**

* `config` holds the provider, model, parameters, instructions or messages, and the tool definitions from `generation`.
* `userInput` is the rendered row input.
* `toolHandlers` maps each tool key to the implementation or provider-native tool passed in `tools`.
* `variables` holds the row variables plus the rendered `input` and `expectedOutput`.
* `history` contains the message history when the handler is used in an agent or tool-calling flow.

Return an object with `output`. Add `usage` with `input`, `output`, and `total` token counts when the provider reports them.

LaunchDarkly records the usage on the row. To test a handler that calls no model, read [Run your first evaluation in Python](#run-your-first-evaluation-in-python) or [Run your first evaluation in Node.js](#run-your-first-evaluation-in-node-js).

## Install the SDK

The functions for running evaluations ship in the AI SDK package for your language: `launchdarkly-ai-server` for Python and `@launchdarkly/ai-node` for Node.js. You also install one handler package for each model provider you generate with.

In Python, also install the base `launchdarkly-server-sdk`. The examples in this topic let `init_evaluations()` create the LaunchDarkly client that sends the results, and that client ships in the base SDK. You can omit it if your application already initializes a LaunchDarkly client and passes it to the SDK, or if the base SDK does not run on your platform. To learn more, read [Bring your own LaunchDarkly client](https://launchdarkly.com/docs/sdk/ai/python#bring-your-own-launchdarkly-client). In Node.js, `@launchdarkly/ai-node` includes the base Node.js SDK. Do not install the base Node.js SDK separately.

Install the SDK and the handler package for the provider you chose in the previous section. For example, to install the SDK and a handler:

**Python SDK**

```bash title="Shell (Python)" lines wrap theme={null}
pip install launchdarkly-server-sdk launchdarkly-ai-server <YOUR-PROVIDER-PACKAGE> python-dotenv
```

**Node.js SDK**

```bash title="Shell (Node.js)" lines wrap theme={null}
npm install @launchdarkly/ai-node <YOUR-PROVIDER-PACKAGE> dotenv
```

## How verdicts work

A criterion is a judge or scorer that grades each row. Each criterion has a `threshold`, which is the score a row must meet to pass that criterion.

For a detailed explanation of how LaunchDarkly decides the verdicts, read [How verdicts work](https://launchdarkly.com/docs/home/agentcontrol/evaluations-from-code.md#how-verdicts-work).

## AI SDKs

This feature is available for the following AI SDKs:

* [Python AI](#python-ai)
* [Node.js (server-side) AI](#node-js-server-side-ai)

### Python AI

> **Expand Python AI SDK code samples**
>
> #### Run your first evaluation in Python
>
> To run your first evaluation, complete the following steps in order.
>
> ##### Verify your setup in Python
>
> Before you make any model calls, check that your credentials, dataset key, and event path work. The following script uses a custom handler that calls no model and returns a fixed output. It needs only your LaunchDarkly credentials. No provider key is required. Here is the script:
>
> ```python title="Python AI SDK" lines wrap theme={null}
> import asyncio
> import os
> import sys
> from datetime import UTC, datetime
> from typing import Any
>
> from dotenv import load_dotenv
> from launchdarkly_ai_server import create_handler, init_evaluations, shutdown
>
>
> def required(name: str) -> str:
>     value = os.getenv(name, "").strip()
>     if not value:
>         raise RuntimeError(f"Set {name} in the environment or .env")
>     return value
>
>
> async def deterministic_handler(
>     config: dict[str, Any],
>     user_input: str | None = None,
>     tool_handlers: dict[str, Any] | None = None,
>     variables: dict[str, Any] | None = None,
> ) -> dict[str, Any]:
>     # Return a fixed output. A real handler calls the model here.
>     return {"output": f"smoke response: {user_input or ''}"}
>
>
> deterministic_handler = create_handler(
>     ("custom", "completion"),
>     deterministic_handler,
> )
>
>
> async def main() -> int:
>     load_dotenv()
>     required("LD_API_TOKEN")
>     required("LD_SDK_KEY")
>     project_key = required("LD_PROJECT_KEY")
>     dataset_key = required("LD_DATASET_KEY")
>     unique = datetime.now(UTC).strftime("%Y%m%d-%H%M%S-%f")
>
>     evals = init_evaluations(project_key=project_key)
>     try:
>         result = await evals.run(
>             key=f"verify-setup-{unique}",
>             dataset=dataset_key,
>             handler=deterministic_handler,
>             generation={
>                 "provider": "custom",
>                 "model": "deterministic-smoke-handler",
>                 "mode": "completion",
>                 "instructions": "Return a deterministic smoke-test response.",
>             },
>             concurrency=2,
>         )
>         print(f"url={result.url}")
>         print(f"summary={result.summary}")
>         return 0 if result.passed else 1
>     finally:
>         await shutdown()
>
>
> if __name__ == "__main__":
>     sys.exit(asyncio.run(main()))
> ```
>
> Save the script as `verify_setup.py` and run it:
>
> ```bash title="Shell (Python)" lines wrap theme={null}
> python verify_setup.py
> ```
>
> If the script prints a URL, your credentials, dataset key, and event path work. If it fails or times out, read [Troubleshooting](#troubleshooting).
>
> ##### Run the evaluation in Python
>
> The following script runs every row of the dataset through OpenAI and reports the results to LaunchDarkly. It uses no criteria, which makes it a generation-only run. Every row that produces an output passes. Here is the script:
>
> ```python title="Python AI SDK" lines wrap theme={null}
> import asyncio
> import os
> import sys
> from datetime import UTC, datetime
>
> from dotenv import load_dotenv
> from launchdarkly_ai_openai_messages import create_openai_messages_handler
> from launchdarkly_ai_server import init_evaluations, shutdown
>
> MODEL = "gpt-4o"
>
>
> def required(name: str) -> str:
>     value = os.getenv(name, "").strip()
>     if not value:
>         raise RuntimeError(f"Set {name} in the environment or .env")
>     return value
>
>
> async def main() -> int:
>     load_dotenv()
>     required("LD_API_TOKEN")
>     required("LD_SDK_KEY")
>     required("OPENAI_API_KEY")
>     project_key = required("LD_PROJECT_KEY")
>     dataset_key = required("LD_DATASET_KEY")
>     unique = datetime.now(UTC).strftime("%Y%m%d-%H%M%S-%f")
>
>     evals = init_evaluations(project_key=project_key)
>     try:
>         result = await evals.run(
>             key=f"first-eval-{unique}",  # Must be unique per run
>             dataset=dataset_key,
>             handler=create_openai_messages_handler(),
>             generation={
>                 "provider": "OpenAI",
>                 "model": MODEL,
>                 "mode": "completion",
>                 "parameters": {"temperature": 0},
>                 "instructions": "Answer the user's question accurately and concisely.",
>             },
>             concurrency=5,
>         )
>         print(f"run_id={result.run_id}")
>         print(f"url={result.url}")
>         print(f"summary={result.summary}")
>         print(f"stored_verdict={'passed' if result.passed else 'failed'}")
>         return 0 if result.passed else 1
>     finally:
>         await shutdown()
>
>
> if __name__ == "__main__":
>     sys.exit(asyncio.run(main()))
> ```
>
> Save the script as `run_first_eval.py` and run it:
>
> ```bash title="Shell (Python)" lines wrap theme={null}
> python run_first_eval.py
> ```
>
> If the run times out or no results appear in LaunchDarkly, read [Troubleshooting](#troubleshooting).
>
> ##### Review your results in Python
>
> The script prints four lines: the run ID, a URL, the summary, and whether the run passed. Here is an example:
>
> ```text title="Python AI SDK" lines wrap theme={null}
> run_id=22222222-2222-2222-2222-222222222222
> url=https://app.launchdarkly.com/projects/my-project/ai/evaluations/11111111-1111-1111-1111-111111111111/runs/22222222-2222-2222-2222-222222222222
> summary=RunSummary(total_rows=10, passed_rows=10, failed_rows=0, error_rows=0, pending_rows=0)
> stored_verdict=passed
> ```
>
> Open the URL to review the run in LaunchDarkly. You can also find the run from the **Evaluations** page. To learn how, read [Review evaluation runs](https://launchdarkly.com/docs/home/agentcontrol/evaluations-from-code.md#review-evaluation-runs).
>
> The `run_id` and `stored_verdict` labels come from the script. The result object that the SDK returns has `run_id`, `url`, `summary`, and `passed`. The script prints `result.summary` directly. This means the summary uses the dataclass representation with snake\_case field names, not JSON. The row counts vary by run.
>
> The terminal shows a short result that is useful in CI, and LaunchDarkly shows the detailed result. The process exits with code `0` when `result.passed` is true, and with `1` otherwise.
>
> The run URL has the following format:
>
> ```text title="Python AI SDK" lines wrap theme={null}
> {ui_base_uri}/projects/{project_key}/ai/evaluations/{evaluation_id}/runs/{run_id}
> ```
>
> The default base URI is `https://app.launchdarkly.com`. To use another LaunchDarkly host, set `LD_UI_BASE_URI`, or pass `ui_base_uri` to `init_evaluations()`. The SDK URL-encodes the project key, evaluation ID, and run ID in the path. Here is an example:
>
> ```python title="Python AI SDK" lines wrap theme={null}
> evals = init_evaluations(
>     project_key=project_key,
>     ui_base_uri="https://app.launchdarkly.com",
> )
> ```
>
> Keep the following in mind about `run()`:
>
> * Every call creates a new evaluation. Use a new `key` for each new run, such as one that includes a timestamp.
> * `dataset` is the dataset key from the Library, not its ID.
> * `generation.mode` is required when you provide `generation` directly. Use `completion` with a messages handler and `agent` with an agent handler.
> * `generation.instructions` is one system message, and `generation.messages` is a full message list. Do not pass both.
> * `concurrency` bounds how many rows run at once. The default is 10.
> * Large datasets might need `poll_timeout_seconds` above the default of 180.
>
> #### Use an inline dataset in Python
>
> An inline dataset is a test set that your script brings with it. Instead of passing a dataset key, you pass the dataset values directly from your code. To learn more, read [Define datasets and tools in code](https://launchdarkly.com/docs/home/agentcontrol/evaluations-from-code.md#define-datasets-and-tools-in-code).
>
> Inline dataset rows use camelCase field names, such as `expectedOutput`. Uploaded dataset files and `DatasetRow` use `expected_output` instead. Here is an example:
>
> ```python title="Python AI SDK" lines wrap theme={null}
> from launchdarkly_ai_openai_messages import create_openai_messages_handler
>
> inline_dataset = [
>     {
>         "input": "What is the status of order {{order_id}}?",
>         "expectedOutput": "Order A19 has shipped and is refund eligible.",
>         "variables": {"order_id": "A19"},
>         "metadata": {"expects_tool_call": True},
>     },
> ]
>
> result = await evals.run(
>     key=f"inline-dataset-{unique}",
>     dataset=inline_dataset,
>     handler=create_openai_messages_handler(),
>     generation={
>         "provider": "OpenAI",
>         "model": "gpt-4o",
>         "mode": "completion",
>         "instructions": "Answer accurately.",
>     },
> )
> ```
>
> The inline dataset must be non-empty. Each row accepts only `input`, `expectedOutput`, `variables`, `metadata`, and `rowIdx`. The SDK rejects any other field. `input` must be a non-empty string. `expectedOutput` is optional and must be a string when supplied. `variables` and `metadata` must be JSON-serializable objects. `rowIdx` is optional and, when supplied, must match the row's zero-based position.
>
> #### Score rows in Python
>
> You can score rows with scorers, LaunchDarkly judges, or both in the same run. To learn how scores become verdicts, read [How verdicts work](#how-verdicts-work).
>
> A `Scorer` wraps a local function. The function receives the `DatasetRow` and the generated output. It returns a finite number from 0 to 1. Nothing else is accepted. A yes-or-no check returns `1.0` or `0.0`. A row passes a scorer when its score meets the `threshold`.
>
> `DatasetRow` exposes `row_index`, `input`, `expected_output`, `variables`, and `metadata`. The rendered input and expected output are also available in `variables`. Here is an example of a scorer:
>
> ```python title="Python AI SDK" lines wrap theme={null}
> from launchdarkly_ai_server import DatasetRow, Scorer
>
>
> def score_expected_text(row: DatasetRow, output: str | None) -> float:
>     """Return 1.0 when the expected output appears in the generated output."""
>     if not row.expected_output or not output:
>         return 0.0
>     expected = " ".join(row.expected_output.casefold().split())
>     generated = " ".join(output.casefold().split())
>     return 1.0 if expected in generated else 0.0
>
>
> scorer = Scorer(
>     name="expected-output-in-answer",
>     fn=score_expected_text,
>     threshold=1.0,
> )
> ```
>
> A `Judge` references a judge config by key. The SDK uses your SDK key to resolve the judge once per run, before it creates any records. A missing or disabled judge fails the run up front. Here is an example of a judge:
>
> ```python title="Python AI SDK" lines wrap theme={null}
> from launchdarkly_ai_server import Judge
>
> JUDGE_KEY = "test-accuracy"
>
> judge = Judge(key=JUDGE_KEY, threshold=0.8)
> ```
>
> To use judges and scorers in the same run, pass both in `criteria`. The following example scores every row with a LaunchDarkly judge and a local scorer:
>
> ```python title="Python AI SDK" lines wrap theme={null}
> from launchdarkly_ai_openai_messages import create_openai_messages_handler
> from launchdarkly_ai_server import Criterion, Judge, Scorer
>
> MODEL = "gpt-4o"
> JUDGE_KEY = "test-accuracy"
>
>
> def score_nonempty_output(row: object, output: str | None) -> float:
>     """Return 1.0 when the output has text."""
>     return 1.0 if output and output.strip() else 0.0
>
>
> criteria: list[Criterion] = [
>     Judge(key=JUDGE_KEY),
>     Scorer(name="nonempty-output", fn=score_nonempty_output, threshold=1.0),
> ]
>
> result = await evals.run(
>     key=f"judges-and-scorers-{unique}",
>     dataset=dataset_key,
>     handler=create_openai_messages_handler(),
>     generation={
>         "provider": "OpenAI",
>         "model": MODEL,
>         "mode": "completion",
>         "parameters": {"temperature": 0},
>         "instructions": "Answer the user's question accurately and concisely.",
>     },
>     criteria=criteria,
>     concurrency=5,
> )
> ```
>
> Judge keys and scorer names share one namespace. They must be unique within a run, without regard to case.
>
> Because a judge is a model call, one of your handlers must serve the judge's provider. If the judge uses a different provider than generation, pass both handlers in `handlers`. The SDK selects the handler whose provider and mode match the generation or judge. Pass exactly one of `handler` and `handlers`. To learn more, read [Judges and scorers](https://launchdarkly.com/docs/home/agentcontrol/evaluations-from-code.md#judges-and-scorers).
>
> For example, to generate with OpenAI and grade with Anthropic:
>
> ```python title="Python AI SDK" lines wrap theme={null}
> from launchdarkly_ai_claude_messages import create_claude_messages_handler
> from launchdarkly_ai_openai_messages import create_openai_messages_handler
> from launchdarkly_ai_server import Judge
>
> JUDGE_KEY = "test-accuracy"
>
> judge = Judge(key=JUDGE_KEY, threshold=0.8)
>
> result = await evals.run(
>     key=f"cross-provider-judges-{unique}",
>     dataset=dataset_key,
>     handlers=[
>         create_openai_messages_handler(),
>         create_claude_messages_handler(),
>     ],
>     generation={
>         "provider": "OpenAI",
>         "model": MODEL,
>         "mode": "completion",
>         "instructions": "Answer the user's question accurately and concisely.",
>     },
>     criteria=[judge],
> )
> ```
>
> The judge config determines which provider the judge uses. The handler pool supplies the matching handler.
>
> #### Judge the tool trajectory in Python
>
> A judge can grade the tool trajectory, which is how the agent reached its answer. The SDK records every tool call the row makes and renders it into `{{message_history}}` between the row input and the output. To learn more, read [Tool trajectory](https://launchdarkly.com/docs/home/agentcontrol/evaluations-from-code.md#tool-trajectory).
>
> Scorers cannot see the trajectory, because a scorer receives only the row and the output. Use a judge for trajectory assertions.
>
> To judge the tool trajectory, complete the following steps in order.
>
> ##### Write the tool implementation in Python
>
> The function receives the parsed arguments and returns a JSON-serializable value. Here is an example:
>
> ```python title="Python AI SDK" lines wrap theme={null}
> from collections.abc import Mapping
> from typing import Any
>
> _ORDERS = {
>     "A19": {"status": "shipped", "refund_eligible": True},
>     "B20": {"status": "processing", "refund_eligible": False},
> }
>
>
> def lookup_order(arguments: Mapping[str, Any]) -> dict[str, str | bool]:
>     """Look up an order by its id."""
>     order_id = arguments.get("order_id")
>     if not isinstance(order_id, str) or not order_id.strip():
>         raise ValueError("lookup_order requires a non-empty string order_id")
>     order = _ORDERS.get(order_id)
>     if order is None:
>         return {"status": "not_found", "refund_eligible": False}
>     return dict(order)
> ```
>
> ##### Make the tool available to the run in Python
>
> Either create the tool in the Library, or define it inline.
>
> To use a Library tool, create a tool in the Library with the key `lookup_order` and the same schema as the inline example in this step. The argument names in the schema must match the arguments your implementation reads. In your script, resolve the tool with `await evals.tools.get()`, providing the tool key and its implementation. `evals.tools.get()` supplies the Library schema and description.
>
> To define the tool inline instead, construct an `EvalTool` with the key, the implementation, the schema, and the description. To learn more, read [Define datasets and tools in code](https://launchdarkly.com/docs/home/agentcontrol/evaluations-from-code.md#define-datasets-and-tools-in-code). Here is an example:
>
> ```python title="Python AI SDK" lines wrap theme={null}
> from launchdarkly_ai_server import EvalTool
>
> inline_tool = EvalTool(
>     key="lookup_order",
>     implementation=lookup_order,
>     schema={
>         "type": "object",
>         "properties": {"order_id": {"type": "string"}},
>         "required": ["order_id"],
>     },
>     description="Look up an order by its ID.",
> )
> ```
>
> The `tools` argument receives the `EvalTool`:
>
> ```python title="Python AI SDK" lines wrap theme={null}
> result = await evals.run(
>     key=f"inline-tool-{unique}",
>     dataset=inline_dataset,
>     handler=create_openai_messages_handler(),
>     tools=[inline_tool],
>     generation={
>         "provider": "OpenAI",
>         "model": "gpt-4o",
>         "mode": "completion",
>         "instructions": "Use lookup_order when an order ID is present.",
>     },
> )
> ```
>
> Inline tool keys must be lowercase and unique within a run. The schema must be a JSON object. To move a tool into the Library, create the corresponding Library tool and change the code to use `evals.tools.get()`. The tool implementation remains in your application.
>
> ##### Build a dataset with rows that need the tool in Python
>
> A trajectory judge can only grade rows that have a reason to call the tool. The example dataset has five rows:
>
> * Two rows that need one call each, for orders A19 and B20.
> * One row for an unknown order, to grade whether the agent reports the miss instead of inventing a status.
> * One row that needs two calls, to grade ordering and that both results reach the answer.
> * One control row that needs no tool, to keep the judge honest about unused tools.
>
> Upload the dataset to the Library as a JSONL file. Here is an example dataset:
>
> ```json title="Python AI SDK (JSONL dataset file)" lines wrap theme={null}
> {"input": "What is the status of order {{order_id}}, and can I get a refund?", "expected_output": "Order A19 has shipped and is refund eligible.", "variables": {"order_id": "A19"}, "metadata": {"expects_tool_call": true}}
> {"input": "Customer is asking about order {{order_id}}. Can they get their money back?", "expected_output": "Order B20 is still processing and is not refund eligible.", "variables": {"order_id": "B20"}, "metadata": {"expects_tool_call": true}}
> {"input": "Look up order {{order_id}} and tell me whether it shipped.", "expected_output": "Order Z99 could not be found.", "variables": {"order_id": "Z99"}, "metadata": {"expects_tool_call": true}}
> {"input": "Compare orders {{first_order}} and {{second_order}}: which one can be refunded?", "expected_output": "A19 is refund eligible; B20 is not.", "variables": {"first_order": "A19", "second_order": "B20"}, "metadata": {"expects_tool_call": true}}
> {"input": "What is your refund policy in general?", "expected_output": "A general policy explanation with no order lookup.", "variables": {}, "metadata": {"expects_tool_call": false}}
> ```
>
> ##### Create the judge and write its rubric in Python
>
> Create a judge in LaunchDarkly. Put only the grading criteria in the system prompt. Do not add `{{message_history}}` to the system prompt, because the judge config's default assistant message already carries it. The SDK appends the JSON score format instructions. Do not restate them. Here is an example system prompt:
>
> ```text title="Judge system prompt" lines wrap theme={null}
> You are grading an assistant's TOOL USE, not its prose.
>
> The message history shows the user's request, the tools the assistant
> could call, the calls it actually made with their arguments and each
> result or error, and its final answer.
>
> Score 1.0 when the tool use was right for the request. That INCLUDES
> calling no tool at all when the request did not need one: an available
> tool left unused is not a mistake, and a request that needs no tool
> must not be penalized for having none.
>
> Score 0.0 when the assistant needed a tool and did not call it, called
> the wrong tool, passed arguments the request does not support, ignored
> what a tool returned, or repeated a failing call without adapting.
>
> Score in between for partial credit.
> ```
>
> Keep the clause about an unused tool. Without it, the judge fails every row whose correct behavior is to call nothing.
>
> ##### Run the trajectory evaluation in Python
>
> Here is an example:
>
> ```python title="Python AI SDK" lines wrap theme={null}
> from launchdarkly_ai_openai_messages import create_openai_messages_handler
> from launchdarkly_ai_server import Judge
>
> MODEL = "gpt-4o"
> TOOL_KEY = "lookup_order"
> TRAJECTORY_JUDGE_KEY = "tool-trajectory-relevance"
> JUDGE_THRESHOLD = 0.8
>
> judge = Judge(key=TRAJECTORY_JUDGE_KEY, threshold=JUDGE_THRESHOLD)
>
> library_tool = await evals.tools.get(
>     TOOL_KEY,
>     implementation=lookup_order,
> )
>
> result = await evals.run(
>     key=f"tool-trajectory-judge-{unique}",
>     dataset=dataset_key,
>     handler=create_openai_messages_handler(),
>     tools=[library_tool],
>     generation={
>         "provider": "OpenAI",
>         "model": MODEL,
>         "mode": "completion",
>         "parameters": {"temperature": 0},
>         "instructions": (
>             f"Use the {TOOL_KEY} tool whenever an order identifier is "
>             "present. Base the final answer on the tool result and answer "
>             "concisely."
>         ),
>     },
>     criteria=[judge],
>     concurrency=2,
> )
> ```
>
> For a row that called one tool, `{{message_history}}` renders like this:
>
> ```text title="Python AI SDK" lines wrap theme={null}
> Use lookup_order for A19
>
> Tools available: lookup_order
> Tool calls made while producing the response, in order:
> 1. lookup_order
>    arguments: {"order_id":"A19"}
>    result: {"refund_eligible":true,"status":"shipped"}
>
> Order A19 is shipped and refund eligible.
> ```
>
> Keep the following in mind when you design a rubric:
>
> * `Tools available` lists what the model could call. A rubric can use it to grade tool selection.
> * A row with tools that called none of them reads `No tool calls were made while producing the response.`
> * A run with no tools adds no trajectory block. Do not point a trajectory judge at a dataset with no tool rows.
> * A failed call shows `error:` in place of `result:`. This is what makes recovery gradeable.
> * Provider built-in tools run inside the provider. They never appear in the trajectory.
> * The trajectory is capped per row. To learn about the limits, read [Limits](#limits).

### Node.js (server-side) AI

> **Expand Node.js (server-side) AI SDK code samples**
>
> #### Run your first evaluation in Node.js
>
> To run your first evaluation, complete the following steps in order.
>
> ##### Verify your setup in Node.js
>
> Before you make any model calls, check that your credentials, dataset key, and event path work. The following script uses a custom handler that calls no model and returns a fixed output. It needs only your LaunchDarkly credentials. No provider key is required. Here is the script:
>
> ```typescript title="Node.js (server-side) AI SDK" lines wrap theme={null}
> import 'dotenv/config';
> import { createHandler, initEvaluations, shutdown } from '@launchdarkly/ai-node';
>
> const deterministicHandler = createHandler(
>   ['custom', 'messages'],
>   async (
>     config,
>     userInput,
>     toolHandlers,
>     variables,
>     history,
>   ) => ({
>     output: `smoke response: ${userInput ?? ''}`,
>   }),
> );
>
> const required = (name) => {
>   const value = process.env[name]?.trim();
>   if (!value) {
>     throw new Error(`Set ${name} in the environment or .env`);
>   }
>   return value;
> };
>
> const projectKey = required('LD_PROJECT_KEY');
> const datasetKey = required('LD_DATASET_KEY');
> const evals = initEvaluations({ projectKey });
> const unique = new Date().toISOString().replace(/[-:.TZ]/g, '');
>
> try {
>   const result = await evals.run({
>     key: `verify-setup-${unique}`,
>     dataset: datasetKey,
>     handler: deterministicHandler,
>     generation: {
>       provider: 'custom',
>       model: 'deterministic-smoke-handler',
>       mode: 'completion',
>       messages: [
>         { role: 'system', content: 'Return a deterministic smoke-test response.' },
>       ],
>     },
>     concurrency: 2,
>   });
>
>   console.log(`url=${result.url}`);
>   console.log(`summary=${JSON.stringify(result.summary)}`);
>   process.exitCode = result.passed ? 0 : 1;
> } finally {
>   await shutdown();
> }
> ```
>
> Save the script as `verify_setup.mjs` and run it:
>
> ```bash title="Shell (Node.js)" lines wrap theme={null}
> node verify_setup.mjs
> ```
>
> If the script prints a URL, your credentials, dataset key, and event path work. If it fails or times out, read [Troubleshooting](#troubleshooting).
>
> ##### Run the evaluation in Node.js
>
> The following script runs every row of the dataset through OpenAI and reports the results to LaunchDarkly. It uses no criteria, which makes it a generation-only run. Every row that produces an output passes. Here is the script:
>
> ```typescript title="Node.js (server-side) AI SDK" lines wrap theme={null}
> import 'dotenv/config';
> import { createOpenAIHandler } from '@launchdarkly/ai-openai-messages';
> import { initEvaluations, shutdown } from '@launchdarkly/ai-node';
>
> const MODEL = 'gpt-4o';
>
> const required = (name) => {
>   const value = process.env[name]?.trim();
>   if (!value) {
>     throw new Error(`Set ${name} in the environment or .env`);
>   }
>   return value;
> };
>
> const projectKey = required('LD_PROJECT_KEY');
> const datasetKey = required('LD_DATASET_KEY');
> const evals = initEvaluations({ projectKey });
> const unique = new Date().toISOString().replace(/[-:.TZ]/g, '');
>
> try {
>   const result = await evals.run({
>     key: `first-eval-${unique}`,
>     dataset: datasetKey,
>     handler: createOpenAIHandler(),
>     generation: {
>       provider: 'OpenAI',
>       model: MODEL,
>       mode: 'completion',
>       parameters: { temperature: 0 },
>       messages: [
>         {
>           role: 'system',
>           content: "Answer the user's question accurately and concisely.",
>         },
>       ],
>     },
>     concurrency: 5,
>   });
>
>   console.log(`run_id=${result.runId}`);
>   console.log(`url=${result.url}`);
>   console.log(`summary=${JSON.stringify(result.summary)}`);
>   console.log(`stored_verdict=${result.passed ? 'passed' : 'failed'}`);
>   process.exitCode = result.passed ? 0 : 1;
> } finally {
>   await shutdown();
> }
> ```
>
> Save the script as `run_first_eval.mjs` and run it:
>
> ```bash title="Shell (Node.js)" lines wrap theme={null}
> node run_first_eval.mjs
> ```
>
> If the run times out or no results appear in LaunchDarkly, read [Troubleshooting](#troubleshooting).
>
> ##### Review your results in Node.js
>
> The script prints four lines: the run ID, a URL, the summary, and whether the run passed. Here is an example:
>
> ```text title="Node.js (server-side) AI SDK" lines wrap theme={null}
> run_id=22222222-2222-2222-2222-222222222222
> url=https://app.launchdarkly.com/projects/my-project/ai/evaluations/11111111-1111-1111-1111-111111111111/runs/22222222-2222-2222-2222-222222222222
> summary={"totalRows":10,"passedRows":10,"failedRows":0,"errorRows":0,"pendingRows":0}
> stored_verdict=passed
> ```
>
> Open the URL to review the run in LaunchDarkly. You can also find the run from the **Evaluations** page. To learn how, read [Review evaluation runs](https://launchdarkly.com/docs/home/agentcontrol/evaluations-from-code.md#review-evaluation-runs).
>
> The `run_id` and `stored_verdict` labels come from the script. The result object that the SDK returns has `runId`, `url`, `summary`, and `passed`. The script prints the summary as JSON with camelCase field names. The row counts vary by run.
>
> The terminal shows a short result that is useful in CI, and LaunchDarkly shows the detailed result. The process exits with code `0` when `result.passed` is true, and with `1` otherwise.
>
> The run URL has the following format:
>
> ```text title="Node.js (server-side) AI SDK" lines wrap theme={null}
> {uiBaseUri}/projects/{projectKey}/ai/evaluations/{evaluationId}/runs/{runId}
> ```
>
> The default base URI is `https://app.launchdarkly.com`. To use another LaunchDarkly host, set `LD_UI_BASE_URI`, or pass `uiBaseUri`. The SDK URL-encodes the project key, evaluation ID, and run ID in the path.
>
> Keep the following in mind about `run()`:
>
> * Every call creates a new evaluation. Use a new `key` for each new run, such as one that includes a timestamp.
> * `dataset` is the dataset key from the Library, not its ID.
> * `generation.mode` is required when you provide `generation` directly. Use `completion` with a messages handler and `agent` with an agent handler.
> * Use `generation.messages` with a messages handler and `generation.instructions` with an agent handler. Do not pass both.
> * `concurrency` bounds how many rows run at once. The default is 10.
> * Large datasets might need `pollTimeoutSeconds` above the default of 180.
>
> #### Use an inline dataset in Node.js
>
> An inline dataset is a test set that your script brings with it. Instead of passing a dataset key, you pass the dataset values directly from your code. To learn more, read [Define datasets and tools in code](https://launchdarkly.com/docs/home/agentcontrol/evaluations-from-code.md#define-datasets-and-tools-in-code).
>
> Inline dataset rows use camelCase field names, such as `expectedOutput`. Uploaded dataset files use `expected_output` instead. Here is an example:
>
> ```typescript title="Node.js (server-side) AI SDK" lines wrap theme={null}
> const inlineDataset = [
>   {
>     input: 'What is the status of order {{orderId}}?',
>     expectedOutput: 'Order A19 has shipped and is refund eligible.',
>     variables: { orderId: 'A19' },
>     metadata: { expectsToolCall: true },
>   },
> ];
>
> const result = await evals.run({
>   key: `inline-dataset-${unique}`,
>   dataset: inlineDataset,
>   handler: createOpenAIHandler(),
>   generation: {
>     provider: 'OpenAI',
>     model: 'gpt-4o',
>     mode: 'completion',
>     messages: [{ role: 'system', content: 'Answer accurately.' }],
>   },
> });
> ```
>
> The inline dataset must be non-empty. Each row accepts only `input`, `expectedOutput`, `variables`, `metadata`, and `rowIdx`. `input` must be a non-empty string. `expectedOutput` is optional and must be a string when supplied. `variables` and `metadata` must be JSON-serializable objects. `rowIdx` is optional and, when supplied, must match the row's zero-based position.
>
> #### Score rows in Node.js
>
> You can score rows with scorers, LaunchDarkly judges, or both in the same run. To learn how scores become verdicts, read [How verdicts work](#how-verdicts-work).
>
> A `Scorer` wraps a local function. The function receives the `DatasetRow` and the generated output. It returns a finite number from 0 to 1. Nothing else is accepted. A yes-or-no check returns `1.0` or `0.0`. A row passes a scorer when its score meets the `threshold`.
>
> `DatasetRow` exposes `rowIndex`, `input`, `expectedOutput`, `variables`, and `metadata`. The rendered input and expected output are also available in `variables`. Here is an example of a scorer:
>
> ```typescript title="Node.js (server-side) AI SDK" lines wrap theme={null}
> import { Scorer } from '@launchdarkly/ai-node';
>
> const scoreExpectedText = (row, output) => {
>   if (!row.expectedOutput || !output) {
>     return 0;
>   }
>
>   const expected = row.expectedOutput.trim().toLowerCase();
>   const generated = output.trim().toLowerCase();
>   return generated.includes(expected) ? 1 : 0;
> };
>
> const scorer = new Scorer(
>   'expected-output-in-answer',
>   scoreExpectedText,
>   { threshold: 1.0 },
> );
> ```
>
> A `Judge` references a judge config by key. The SDK uses your SDK key to resolve the judge once per run, before it creates any records. A missing or disabled judge fails the run up front. Here is an example of a judge:
>
> ```typescript title="Node.js (server-side) AI SDK" lines wrap theme={null}
> import { Judge } from '@launchdarkly/ai-node';
>
> const JUDGE_KEY = 'test-accuracy';
>
> const judge = new Judge(JUDGE_KEY, {
>   threshold: 0.8,
> });
> ```
>
> To use judges and scorers in the same run, pass both in `criteria`. The following example scores every row with a LaunchDarkly judge and a local scorer:
>
> ```typescript title="Node.js (server-side) AI SDK" lines wrap theme={null}
> import { createOpenAIHandler } from '@launchdarkly/ai-openai-messages';
> import {
>   Judge,
>   Scorer,
> } from '@launchdarkly/ai-node';
>
> const MODEL = 'gpt-4o';
> const JUDGE_KEY = 'test-accuracy';
>
> const scoreNonemptyOutput = (_row, output) =>
>   output && output.trim() ? 1 : 0;
>
> const criteria = [
>   new Judge(JUDGE_KEY),
>   new Scorer('nonempty-output', scoreNonemptyOutput, { threshold: 1.0 }),
> ];
>
> const result = await evals.run({
>   key: `judges-and-scorers-${unique}`,
>   dataset: datasetKey,
>   handler: createOpenAIHandler(),
>   generation: {
>     provider: 'OpenAI',
>     model: MODEL,
>     mode: 'completion',
>     parameters: { temperature: 0 },
>     messages: [
>       {
>         role: 'system',
>         content: "Answer the user's question accurately and concisely.",
>       },
>     ],
>   },
>   criteria,
>   concurrency: 5,
> });
> ```
>
> Judge keys and scorer names share one namespace. They must be unique within a run, without regard to case.
>
> Because a judge is a model call, one of your handlers must serve the judge's provider. If the judge uses a different provider than generation, pass both handlers in `handlers`. The SDK selects the handler whose provider and mode match the generation or judge. Pass exactly one of `handler` and `handlers`. To learn more, read [Judges and scorers](https://launchdarkly.com/docs/home/agentcontrol/evaluations-from-code.md#judges-and-scorers).
>
> For example, to generate with OpenAI and grade with Anthropic:
>
> ```typescript title="Node.js (server-side) AI SDK" lines wrap theme={null}
> import { createClaudeMessagesHandler } from '@launchdarkly/ai-claude-messages';
> import { createOpenAIHandler } from '@launchdarkly/ai-openai-messages';
>
> const result = await evals.run({
>   key: `cross-provider-judges-${unique}`,
>   dataset: datasetKey,
>   handlers: [
>     createOpenAIHandler(),
>     createClaudeMessagesHandler(),
>   ],
>   generation: {
>     provider: 'OpenAI',
>     model: 'gpt-4o',
>     mode: 'completion',
>     messages: [{ role: 'system', content: 'Answer accurately.' }],
>   },
>   criteria: [new Judge('anthropic-accuracy')],
> });
> ```
>
> The judge config determines which provider the judge uses. The handler pool supplies the matching handler.
>
> #### Judge the tool trajectory in Node.js
>
> A judge can grade the tool trajectory, which is how the agent reached its answer. The SDK records every tool call the row makes and renders it into `{{message_history}}` between the row input and the output. To learn more, read [Tool trajectory](https://launchdarkly.com/docs/home/agentcontrol/evaluations-from-code.md#tool-trajectory).
>
> Scorers cannot see the trajectory, because a scorer receives only the row and the output. Use a judge for trajectory assertions.
>
> To judge the tool trajectory, complete the following steps in order.
>
> ##### Write the tool implementation in Node.js
>
> The function receives the parsed arguments and returns a JSON-serializable value. Here is an example:
>
> ```typescript title="Node.js (server-side) AI SDK" lines wrap theme={null}
> const orders = {
>   A19: { status: 'shipped', refundEligible: true },
>   B20: { status: 'processing', refundEligible: false },
> };
>
> const lookupOrder = async ({ orderId }) => {
>   if (typeof orderId !== 'string' || !orderId.trim()) {
>     throw new Error('lookup_order requires a non-empty string orderId');
>   }
>
>   return orders[orderId] ?? {
>     status: 'not_found',
>     refundEligible: false,
>   };
> };
> ```
>
> ##### Make the tool available to the run in Node.js
>
> Either create the tool in the Library, or define it inline.
>
> To use a Library tool, create a tool in the Library with the key `lookup_order` and the same schema as the inline example in this step. The argument names in the schema must match the arguments your implementation reads. In your script, resolve the tool with `await evals.tools.get()`, providing the tool key and its implementation. `evals.tools.get()` supplies the Library schema and description, and pins the tool version for the run.
>
> To define the tool inline instead, construct an `EvalTool` with the key, the implementation, the schema, and the description. To learn more, read [Define datasets and tools in code](https://launchdarkly.com/docs/home/agentcontrol/evaluations-from-code.md#define-datasets-and-tools-in-code). Here is an example:
>
> ```typescript title="Node.js (server-side) AI SDK" lines wrap theme={null}
> import { EvalTool } from '@launchdarkly/ai-node';
>
> const inlineTool = new EvalTool(
>   'lookup_order',
>   lookupOrder,
>   {
>     type: 'object',
>     properties: { orderId: { type: 'string' } },
>     required: ['orderId'],
>   },
>   'Look up an order by its ID.',
> );
> ```
>
> The `tools` argument receives the `EvalTool`:
>
> ```typescript title="Node.js (server-side) AI SDK" lines wrap theme={null}
> const result = await evals.run({
>   key: `inline-tool-${unique}`,
>   dataset: inlineDataset,
>   handler: createOpenAIHandler(),
>   tools: [inlineTool],
>   generation: {
>     provider: 'OpenAI',
>     model: 'gpt-4o',
>     mode: 'completion',
>     messages: [
>       {
>         role: 'system',
>         content: 'Use lookup_order when an order ID is present.',
>       },
>     ],
>   },
> });
> ```
>
> Inline tool keys must be lowercase and unique within a run. To move a tool into the Library, create the corresponding Library tool and change the code to use `evals.tools.get()`. The tool implementation remains in your application.
>
> ##### Build a dataset with rows that need the tool in Node.js
>
> A trajectory judge can only grade rows that have a reason to call the tool. The example dataset has five rows:
>
> * Two rows that need one call each, for orders A19 and B20.
> * One row for an unknown order, to grade whether the agent reports the miss instead of inventing a status.
> * One row that needs two calls, to grade ordering and that both results reach the answer.
> * One control row that needs no tool, to keep the judge honest about unused tools.
>
> Pass the rows as an inline dataset. Here is an example dataset:
>
> ```typescript title="Node.js (server-side) AI SDK" lines wrap theme={null}
> const trajectoryDataset = [
>   {
>     input: 'What is the status of order {{orderId}}, and can I get a refund?',
>     expectedOutput: 'Order A19 has shipped and is refund eligible.',
>     variables: { orderId: 'A19' },
>     metadata: { expectsToolCall: true },
>   },
>   {
>     input: 'Customer is asking about order {{orderId}}. Can they get their money back?',
>     expectedOutput: 'Order B20 is still processing and is not refund eligible.',
>     variables: { orderId: 'B20' },
>     metadata: { expectsToolCall: true },
>   },
>   {
>     input: 'Look up order {{orderId}} and tell me whether it shipped.',
>     expectedOutput: 'Order Z99 could not be found.',
>     variables: { orderId: 'Z99' },
>     metadata: { expectsToolCall: true },
>   },
>   {
>     input: 'Compare orders {{firstOrder}} and {{secondOrder}}: which one can be refunded?',
>     expectedOutput: 'A19 is refund eligible; B20 is not.',
>     variables: { firstOrder: 'A19', secondOrder: 'B20' },
>     metadata: { expectsToolCall: true },
>   },
>   {
>     input: 'What is your refund policy in general?',
>     expectedOutput: 'A general policy explanation with no order lookup.',
>     variables: {},
>     metadata: { expectsToolCall: false },
>   },
> ];
> ```
>
> ##### Create the judge and write its rubric in Node.js
>
> Create a judge in LaunchDarkly. Put only the grading criteria in the system prompt. Do not add `{{message_history}}` to the system prompt, because the judge config's default assistant message already carries it. The SDK appends the JSON score format instructions. Do not restate them. Here is an example system prompt:
>
> ```text title="Judge system prompt" lines wrap theme={null}
> You are grading an assistant's TOOL USE, not its prose.
>
> The message history shows the user's request, the tools the assistant
> could call, the calls it actually made with their arguments and each
> result or error, and its final answer.
>
> Score 1.0 when the tool use was right for the request. That INCLUDES
> calling no tool at all when the request did not need one: an available
> tool left unused is not a mistake, and a request that needs no tool
> must not be penalized for having none.
>
> Score 0.0 when the assistant needed a tool and did not call it, called
> the wrong tool, passed arguments the request does not support, ignored
> what a tool returned, or repeated a failing call without adapting.
>
> Score in between for partial credit.
> ```
>
> Keep the clause about an unused tool. Without it, the judge fails every row whose correct behavior is to call nothing.
>
> ##### Run the trajectory evaluation in Node.js
>
> Here is an example:
>
> ```typescript title="Node.js (server-side) AI SDK" lines wrap theme={null}
> const trajectoryJudge = new Judge('tool-trajectory-relevance', {
>   threshold: 0.8,
> });
>
> const libraryTool = await evals.tools.get('lookup_order', {
>   implementation: lookupOrder,
> });
>
> const result = await evals.run({
>   key: `tool-trajectory-judge-${unique}`,
>   dataset: trajectoryDataset,
>   handler: createOpenAIHandler(),
>   tools: [libraryTool],
>   generation: {
>     provider: 'OpenAI',
>     model: 'gpt-4o',
>     mode: 'completion',
>     messages: [
>       {
>         role: 'system',
>         content:
>           'Use lookup_order whenever an order identifier is present. ' +
>           'Base the final answer on the tool result and answer concisely.',
>       },
>     ],
>   },
>   criteria: [trajectoryJudge],
>   concurrency: 2,
> });
> ```
>
> For a row that called one tool, `{{message_history}}` renders like this:
>
> ```text title="Node.js (server-side) AI SDK" lines wrap theme={null}
> Use lookup_order for A19
>
> Tools available: lookup_order
> Tool calls made while producing the response, in order:
> 1. lookup_order
>    arguments: {"orderId":"A19"}
>    result: {"refundEligible":true,"status":"shipped"}
>
> Order A19 is shipped and refund eligible.
> ```
>
> Keep the following in mind when you design a rubric:
>
> * `Tools available` lists what the model could call. A rubric can use it to grade tool selection.
> * A row with tools that called none of them reads `No tool calls were made while producing the response.`
> * A run with no tools adds no trajectory block. Do not point a trajectory judge at a dataset with no tool rows.
> * A failed call shows `error:` in place of `result:`. This is what makes recovery gradeable.
> * Provider built-in tools run inside the provider. They never appear in the trajectory.
> * The trajectory is capped per row. To learn about the limits, read [Limits](#limits).

## Run evaluations in CI

The script in [Run your first evaluation in Python](#run-your-first-evaluation-in-python) or [Run your first evaluation in Node.js](#run-your-first-evaluation-in-node-js) exits with code `0` when the stored run passes, and with `1` when it does not. Most CI systems treat a non-zero exit code as a failed step. This means the result of a run can fail a build or block a release.

Keep the following in mind when you run in CI:

* **Store credentials as secrets:** Add `LD_API_TOKEN` and `LD_SDK_KEY`, the project key, and any provider key as secrets in your CI system. Pass them to the job as environment variables. Do not commit them.
* **Use a unique run key for each new run:** Every run creates a new evaluation. A timestamp works, and you can also include the CI run number.
* **Retry a failed run with the same key:** To retry a run that failed, run the script again with the same run key.
* **Create Library assets first:** Library datasets, tools, and judges must exist in the project before the job runs. Inline datasets and inline tools live in your code and travel with your repository.
* **Raise limits for large datasets:** Large datasets might need `poll_timeout_seconds` in Python, or `pollTimeoutSeconds` in Node.js, above the default of 180.
* **Open the run URL for details:** The terminal shows a short result, and LaunchDarkly shows the detailed result.

## Options

This section lists the arguments and settings that control a run.

### Run options

Pass the key of the LaunchDarkly project that holds the dataset, tools, and judges when you initialize evaluations. In Python, pass `project_key` to `init_evaluations()`. In Node.js, pass `projectKey` to `initEvaluations()`, or set `LD_PROJECT_KEY`.

The `run()` function accepts the following arguments:

**Python**

<table>
  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`key`</td>
      <td>(Required) The key for the evaluation. Use a new key for each new run. To retry a failed run, reuse its key.</td>
    </tr>

    <tr>
      <td>`dataset`</td>
      <td>(Required) The dataset key from the Library, not its ID, or a list of inline dataset rows.</td>
    </tr>

    <tr>
      <td>`handler`</td>
      <td>(Required unless you pass `handlers`) A LaunchDarkly handler or a custom handler function.</td>
    </tr>

    <tr>
      <td>`generation`</td>
      <td>(Required) The `provider`, `model`, `mode`, `parameters`, and either `instructions` or `messages`. `mode` is required when you provide `generation` directly. Use `instructions` for one system message and `messages` for a full message list. Do not pass both.</td>
    </tr>

    <tr>
      <td>`tools`</td>
      <td>A list of `EvalTool` objects. Use `evals.tools.get()` for Library tools or construct an `EvalTool` for an inline tool.</td>
    </tr>

    <tr>
      <td>`criteria`</td>
      <td>A list of `Judge` and `Scorer` objects. With no criteria, every row that produces an output passes.</td>
    </tr>

    <tr>
      <td>`handlers`</td>
      <td>A list of handlers used when generation and judges require different provider or mode combinations. Pass exactly one of `handler` and `handlers`.</td>
    </tr>

    <tr>
      <td>`concurrency`</td>
      <td>How many rows run at once. Defaults to **10**.</td>
    </tr>

    <tr>
      <td>`poll_interval_seconds`</td>
      <td>How many seconds the SDK waits between checks of the run summary. Defaults to **2 seconds**.</td>
    </tr>

    <tr>
      <td>`poll_timeout_seconds`</td>
      <td>How long the SDK waits for the run to finish. Large datasets might need a higher value. Defaults to **180 seconds**.</td>
    </tr>
  </tbody>
</table>

**Node.js**

<table>
  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`key`</td>
      <td>(Required) The key for the evaluation. Use a new key for each new run. To retry a failed run, reuse its key.</td>
    </tr>

    <tr>
      <td>`dataset`</td>
      <td>(Required) The dataset key from the Library, not its ID, or an array of inline dataset rows.</td>
    </tr>

    <tr>
      <td>`handler`</td>
      <td>(Required unless you pass `handlers`) A LaunchDarkly handler, or a custom handler built with `createHandler()`.</td>
    </tr>

    <tr>
      <td>`generation`</td>
      <td>(Required) The `provider`, `model`, `mode`, `parameters`, and either `messages` or `instructions`. `mode` is `completion` or `agent`, and is required when you provide `generation` directly. Use `messages` with a messages handler and `instructions` with an agent handler. Do not pass both.</td>
    </tr>

    <tr>
      <td>`tools`</td>
      <td>An array of `EvalTool` instances. Use `evals.tools.get()` for Library tools or construct an `EvalTool` for an inline tool.</td>
    </tr>

    <tr>
      <td>`criteria`</td>
      <td>An array of `Judge` and `Scorer` instances. With no criteria, every row that produces an output passes.</td>
    </tr>

    <tr>
      <td>`handlers`</td>
      <td>An array of handlers used when generation and judges require different provider or mode combinations. Pass exactly one of `handler` and `handlers`.</td>
    </tr>

    <tr>
      <td>`concurrency`</td>
      <td>How many rows run at once. Defaults to **10**.</td>
    </tr>

    <tr>
      <td>`pollIntervalSeconds`</td>
      <td>How many seconds the SDK waits between checks of the run summary. Defaults to **2 seconds**.</td>
    </tr>

    <tr>
      <td>`pollTimeoutSeconds`</td>
      <td>How long the SDK waits for the run to finish. Large datasets might need a higher value. Defaults to **180 seconds**.</td>
    </tr>
  </tbody>
</table>

### Scorer settings

A `Scorer` accepts the following settings:

**Python**

<table>
  <thead>
    <tr>
      <th>Setting</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`name`</td>
      <td>(Required) The name of the scorer. Names and judge keys share one namespace and must be unique within a run, without regard to case.</td>
    </tr>

    <tr>
      <td>`fn`</td>
      <td>(Required) The function that receives the `DatasetRow` and the output, and returns a number from 0 to 1.</td>
    </tr>

    <tr>
      <td>`threshold`</td>
      <td>The score a row must meet to pass. Defaults to **1.0**, which means a row passes only on a perfect score. Lower it for graded scorers.</td>
    </tr>

    <tr>
      <td>`success_direction`</td>
      <td>`"higher_is_better"` or `"lower_is_better"`. Use `"lower_is_better"` for a scorer that counts something unwanted. Defaults to **`"higher_is_better"`**.</td>
    </tr>
  </tbody>
</table>

**Node.js**

The `Scorer` constructor takes the name and the function, followed by an options object with the remaining settings.

<table>
  <thead>
    <tr>
      <th>Setting</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`name`</td>
      <td>(Required) The name of the scorer. Names and judge keys share one namespace and must be unique within a run, without regard to case.</td>
    </tr>

    <tr>
      <td>`fn`</td>
      <td>(Required) The function that receives the `DatasetRow` and the output, and returns a number from 0 to 1.</td>
    </tr>

    <tr>
      <td>`threshold`</td>
      <td>The score a row must meet to pass. Defaults to **1.0**, which means a row passes only on a perfect score. Lower it for graded scorers.</td>
    </tr>

    <tr>
      <td>`successDirection`</td>
      <td>`'higher_is_better'` or `'lower_is_better'`. Use `'lower_is_better'` for a scorer that counts something unwanted. Defaults to **`'higher_is_better'`**.</td>
    </tr>
  </tbody>
</table>

### Judge settings

A `Judge` accepts the following settings:

**Python**

<table>
  <thead>
    <tr>
      <th>Setting</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`key`</td>
      <td>(Required) The key of the judge config.</td>
    </tr>

    <tr>
      <td>`threshold`</td>
      <td>The score a row must meet to pass. A row passes when the judge score meets it. Defaults to **0.5**.</td>
    </tr>

    <tr>
      <td>`ground_truth_context`</td>
      <td>Replaces the expected output as the reference the judge grades against.</td>
    </tr>
  </tbody>
</table>

**Node.js**

The `Judge` constructor takes the key, followed by an options object with the remaining settings.

<table>
  <thead>
    <tr>
      <th>Setting</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`key`</td>
      <td>(Required) The key of the judge config.</td>
    </tr>

    <tr>
      <td>`threshold`</td>
      <td>The score a row must meet to pass. A row passes when the judge score meets it. Defaults to **0.5**.</td>
    </tr>

    <tr>
      <td>`groundTruthContext`</td>
      <td>Replaces the expected output as the reference the judge grades against.</td>
    </tr>
  </tbody>
</table>

The success direction of a judge lives on the judge config in LaunchDarkly. You do not set it in code.

## Limits

The following limits apply to evaluation runs that you start from code:

<table>
  <thead>
    <tr>
      <th>Limit</th>
      <th>Value</th>
      <th>What happens when exceeded</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Uploaded dataset file size</td>
      <td>Under 32 MB</td>
      <td>The upload fails validation and LaunchDarkly shows an error.</td>
    </tr>

    <tr>
      <td>Rows in an uploaded dataset</td>
      <td>Fewer than 10,000</td>
      <td>The upload fails validation and LaunchDarkly shows an error.</td>
    </tr>
  </tbody>
</table>

The SDK records up to 50 tool calls per row, and up to 2,000 characters for each set of tool arguments or each result.

For the default `concurrency`, `poll_timeout_seconds`, and `pollTimeoutSeconds` values, read [Run options](#run-options).

## Troubleshooting

This table includes possible errors and their solutions:

**Python**

<table>
  <thead>
    <tr>
      <th>Symptom</th>
      <th>Likely cause</th>
      <th>Resolution</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`ImportError` for `EvalTool`, `create_handler`, or another name this guide imports</td>
      <td>The installed SDK is too old.</td>
      <td>Install `launchdarkly-ai-server` 1.0.0 or later.</td>
    </tr>

    <tr>
      <td>`EvaluationsError: No LaunchDarkly SDK key provided`</td>
      <td>`LD_SDK_KEY` is not set. Every path needs it.</td>
      <td>Set `LD_SDK_KEY` in your environment or `.env` file, then run the script again.</td>
    </tr>

    <tr>
      <td>The run times out and no results appear</td>
      <td>The SDK key is missing or wrong, which means LaunchDarkly ingests no results.</td>
      <td>Set `LD_SDK_KEY` to the server-side SDK key for the environment you want results in.</td>
    </tr>

    <tr>
      <td>`HTTP 401`</td>
      <td>The API access token is missing or invalid.</td>
      <td>Check `LD_API_TOKEN`. Create a new token if the old one expired.</td>
    </tr>

    <tr>
      <td>`HTTP 403`</td>
      <td>The token lacks a dataset, tool, evaluation, or run permission.</td>
      <td>Use the **Writer** role, or update the custom role to allow reading datasets and tools and creating evaluations and evaluation runs in the target project.</td>
    </tr>

    <tr>
      <td>`HTTP 404`</td>
      <td>The project key, dataset key, tool key, or judge key does not exist in the project you passed.</td>
      <td>Check that all of the keys belong to the same project.</td>
    </tr>

    <tr>
      <td>The run fails before it creates any records, and names a tool key</td>
      <td>A tool key you passed in `tools` is not in the Library and was not defined inline.</td>
      <td>Create the tool in the Library with the same key, or define it inline.</td>
    </tr>

    <tr>
      <td>Judge resolution fails before the run starts</td>
      <td>The SDK key is wrong, the judge config does not exist or has been disabled, or your handler cannot serve the judge's provider.</td>
      <td>Check `LD_SDK_KEY`, confirm the judge config exists and is enabled, and confirm your handler can serve the judge's provider. If the judge uses a different provider, include handlers for both providers in `handlers`.</td>
    </tr>

    <tr>
      <td>The run stays pending</td>
      <td>Not every row was accepted, or events did not flush.</td>
      <td>Confirm that every row was accepted and that your script calls `shutdown()` so events flush. Raise `poll_timeout_seconds` for large datasets.</td>
    </tr>

    <tr>
      <td>The trajectory judge fails rows that need no tool</td>
      <td>The rubric penalizes an unused tool.</td>
      <td>Add the unused-tool clause to the rubric, or point the judge at a dataset where every row needs the tool.</td>
    </tr>
  </tbody>
</table>

**Node.js**

<table>
  <thead>
    <tr>
      <th>Symptom</th>
      <th>Likely cause</th>
      <th>Resolution</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>An import of `initEvaluations`, `EvalTool`, or another name this guide imports fails</td>
      <td>The installed SDK is too old.</td>
      <td>Install `@launchdarkly/ai-node` 1.0.0 or later.</td>
    </tr>

    <tr>
      <td>`EvaluationsError` that names `LD_PROJECT_KEY`, `LD_API_TOKEN`, or `LD_SDK_KEY`</td>
      <td>That value is not set. `initEvaluations()` names the first missing value.</td>
      <td>Set the value in your environment or `.env` file, or pass it to `initEvaluations()`, then run the script again.</td>
    </tr>

    <tr>
      <td>The run times out and no results appear</td>
      <td>The SDK key is missing or wrong, which means LaunchDarkly ingests no results.</td>
      <td>Set `LD_SDK_KEY` to the server-side SDK key for the environment you want results in.</td>
    </tr>

    <tr>
      <td>`HTTP 401`</td>
      <td>The API access token is missing or invalid.</td>
      <td>Check `LD_API_TOKEN`. Create a new token if the old one expired.</td>
    </tr>

    <tr>
      <td>`HTTP 403`</td>
      <td>The token lacks a dataset, tool, evaluation, or run permission.</td>
      <td>Use the **Writer** role, or update the custom role to allow reading datasets and tools and creating evaluations and evaluation runs in the target project.</td>
    </tr>

    <tr>
      <td>`HTTP 404`</td>
      <td>The project key, dataset key, tool key, or judge key does not exist in the project you passed.</td>
      <td>Check that all of the keys belong to the same project.</td>
    </tr>

    <tr>
      <td>`generation.mode is required. Set it to 'completion' or 'agent'.`</td>
      <td>`generation` has no `mode`.</td>
      <td>Set `mode` to `completion` for a messages handler, or `agent` for an agent handler.</td>
    </tr>

    <tr>
      <td>The run fails because the generation needs `messages`, not `instructions`, or the reverse</td>
      <td>The prompt field does not match the handler's mode.</td>
      <td>Use `messages` with a messages handler and `instructions` with an agent handler.</td>
    </tr>

    <tr>
      <td>The run fails because a handler does not declare `providesFor`</td>
      <td>A custom handler is a plain function.</td>
      <td>Wrap the handler with `createHandler()` to declare the provider and mode that it serves.</td>
    </tr>

    <tr>
      <td>The run fails before it creates any records, and names a tool key</td>
      <td>A tool key you passed in `tools` is not in the Library and was not defined inline.</td>
      <td>Create the tool in the Library with the same key, or define it inline.</td>
    </tr>

    <tr>
      <td>Judge resolution fails before the run starts</td>
      <td>The SDK key is wrong, the judge config does not exist or has been disabled, or your handler cannot serve the judge's provider.</td>
      <td>Check `LD_SDK_KEY`, confirm the judge config exists and is active, and confirm your handler can serve the judge's provider. If the judge uses a different provider, include handlers for both providers in `handlers`.</td>
    </tr>

    <tr>
      <td>The run stays pending</td>
      <td>Not every row was accepted, or events did not flush.</td>
      <td>Confirm that every row was accepted and that your script calls `shutdown()` so events flush. Raise `pollTimeoutSeconds` for large datasets.</td>
    </tr>

    <tr>
      <td>The trajectory judge fails rows that need no tool</td>
      <td>The rubric penalizes an unused tool.</td>
      <td>Add the unused-tool clause to the rubric, or point the judge at a dataset where every row needs the tool.</td>
    </tr>
  </tbody>
</table>