Evaluations from code - LaunchDarkly

This topic explains how to run evaluations from code with the LaunchDarkly Python AI SDK or Node.js AI SDK. In addition to running evaluations in the playground, you can run them from your own code. This lets you use your own generation code against a dataset, score each output, and send the results to LaunchDarkly. LaunchDarkly stores the results. With the Python and Node.js AI SDKs, you can:

Generate an output for every row of a dataset, using a handler for your model provider or a handler that you write.
Score each output with deterministic scorers that you write in code, with LaunchDarkly judges, or with both.
Grade the tool calls an agent makes to reach its answer, such as which tools it calls, in what order, and with what arguments.
Define datasets and tools inline in your code, or use datasets and tools from the LaunchDarkly Library.
Use the pass or fail result of a run to set your script’s exit code, for example in a CI workflow.

To learn more about the concepts behind this feature, read Evaluations from code.

How evaluations from code work

A single call to run() generates an output for each dataset row and scores it. Then, it returns the run URL, the summary, and whether the run passed. LaunchDarkly derives each row verdict from the scores, and the SDK reads the run summary to report whether the run passed. To learn more, read How evaluations from code work. Keep the following in mind as you write code to run evaluations:

The SDK needs an API access token and an SDK key.
- API access token: Resolves datasets, tools, and judges, creates the evaluation and the run, and polls the summary.
- SDK key: Sends every generation and criterion result to LaunchDarkly as an event. Results reach LaunchDarkly only through this channel.
Every call to run() creates a new evaluation. Use a new key for each new run, such as one that includes a timestamp. To retry a failed run, reuse its key.
If your handler raises an error for a row, LaunchDarkly records the row as ERROR and the other rows continue.
The SDK does not retry a handler for a row that raises an error.

Prerequisites

To run your first evaluation, you need:

A LaunchDarkly project.
One of the following AI SDKs:
- The Python AI SDK launchdarkly-ai-server 1.0.0 or later, with Python 3.12 or later.
- The Node.js (server-side) AI SDK @launchdarkly/ai-node 1.0.0 or later, with Node.js 22 or later.
An API access token and the environment’s server-side SDK key. To learn how to get them, read Get your credentials.
One handler package for the model provider you generate with, and the API key or credentials for that provider. You do not need a provider key to verify your setup. To learn how, read Run your first evaluation in Python or Run your first evaluation in Node.js.
A dataset, either in the Library or defined inline in your code. To learn more, read Datasets.

Depending on how you score rows and what your agent does, you might also need:

A judge, if you score with LaunchDarkly judges. The judge must exist in the project as a judge config. To learn more, read Judges.
A tool definition for each tool your agent calls, in the Library or inline in your code. To learn more, read Tools.

Get your credentials

The SDK needs two credentials: an API access token and an SDK key. In LaunchDarkly, the API access token is called the Eval API key. To learn what each credential does, read How evaluations from code work.

The SDK key is required. Without it, no result is ingested and every run times out. init_evaluations() in Python and initEvaluations() in Node.js raise an error immediately if the key is missing.

Generate your credentials

To generate your credentials in LaunchDarkly:

Click Evaluations in the side navigation.
Click Run evaluations, then choose From code. The “Run evals from code” dialog appears.
Choose how you want to set up:
- Set up with an agent gives you instructions to hand to your agent.
- Set up manually gives you example code for the language and provider you select. The “Connect your code” dialog appears.
(Optional) If you plan to use the credentials in CI or other automation, check Create a service token under Eval API key.
Click Generate keys. The “Finish setup” dialog appears with your new credentials and a .env file.
Copy the credentials, the .env file, and the information specific to your choice:
- Set up with an agent: The instructions for your agent.
- Set up manually: Choose your programming language and preferred provider to generate an install command and a code example.
Click Done.

You can also create an API access token and copy an SDK key yourself. To learn how, read API access tokens and SDK keys. If you create the token yourself, assign a role that allows reading datasets and tools, and creating evaluations and evaluation runs, in the target project. The Writer role works.

Write the .env file

If you generated your credentials in LaunchDarkly, the “Finish setup” dialog gives you a .env file with your credentials that you can download. It does not include your project key or dataset key. Add LD_PROJECT_KEY and LD_DATASET_KEY to it. Otherwise, create a .env file. Keep only secrets and project identifiers in your .env file. Model names, tool keys, judge keys, and thresholds live in your scripts. For security, add the .env to your .gitignore file. Here is an example:

.env file

LD_API_TOKEN=api-...
LD_SDK_KEY=sdk-...
LD_PROJECT_KEY=your-project-key
LD_DATASET_KEY=your-dataset-key

# Only for runs that call OpenAI
OPENAI_API_KEY=sk-...

Add your provider credentials

LaunchDarkly credentials and model provider credentials authorize different things. Your LaunchDarkly credentials let the SDK create the evaluation, report results, and read the run summary. Your provider credentials let your handler call the model. Add your provider’s API key to your .env file, such as OPENAI_API_KEY or ANTHROPIC_API_KEY. If your provider uses other credentials, such as Amazon Bedrock through a LangChain handler or a custom handler, configure them the way that provider’s SDK expects. To learn more, read Choose a provider handler. To write your own handler, read Use a custom handler.

Choose a provider handler

The handler argument to run() accepts any LaunchDarkly handler. Each handler reads the provider, model, mode, parameters, and instructions or messages from the generation argument. Set generation.provider and generation.mode to match the handler. The handler packages for each SDK are:

Python
Node.js

Package	Factory	Use when
`launchdarkly-ai-openai-messages`	`create_openai_messages_handler()`	OpenAI, single-turn or tool-calling chat
`launchdarkly-ai-openai-agents`	`create_openai_agent_handler()`	OpenAI Agents SDK
`launchdarkly-ai-claude-messages`	`create_claude_messages_handler()`	Anthropic Messages API
`launchdarkly-ai-claude-agents`	`create_claude_agents_handler()`	Claude Agent SDK
`launchdarkly-ai-langchain-messages`	`create_langchain_messages_handler()`	Any LangChain chat model
`launchdarkly-ai-langchain-agents`	`create_langchain_agents_handler()`	LangChain agents
Your own function	`create_handler()`	A provider that the SDK has no package for, an internal gateway, or replaying stored outputs. To learn more, read Use a custom handler.

Package	Factory	Use when
`@launchdarkly/ai-openai-messages`	`createOpenAIHandler()`	OpenAI, single-turn or tool-calling chat
`@launchdarkly/ai-openai-agents`	`createOpenAIAgentHandler()`	OpenAI Agents SDK
`@launchdarkly/ai-claude-messages`	`createClaudeMessagesHandler()`	Anthropic Messages API
`@launchdarkly/ai-claude-agents`	`createClaudeAgentsHandler()`	Claude Agent SDK
`@launchdarkly/ai-langchain-messages`	`createLangChainHandler()`	Any LangChain chat model
`@launchdarkly/ai-langchain-agents`	`createLangChainAgentsHandler()`	LangChain agents
Your own function	`createHandler()`	A provider that the SDK has no package for, an internal gateway, or replaying stored outputs. To learn more, read Use a custom handler.

To use a different provider, install its handler package, pass its factory as handler, and set generation.provider and generation.mode to match. For example, to generate with Anthropic:

from launchdarkly_ai_claude_messages import create_claude_messages_handler

result = await evals.run(
    key=f"first-eval-anthropic-{unique}",
    dataset=dataset_key,
    handler=create_claude_messages_handler(),
    generation={
        "provider": "Anthropic",
        "model": "claude-sonnet-4-5",
        "mode": "completion",
        "instructions": "Answer the user's question accurately and concisely.",
    },
)

import { createClaudeMessagesHandler } from '@launchdarkly/ai-claude-messages';

const result = await evals.run({
  key: `first-eval-anthropic-${unique}`,
  dataset: datasetKey,
  handler: createClaudeMessagesHandler(),
  generation: {
    provider: 'Anthropic',
    model: 'claude-sonnet-4-5',
    mode: 'completion',
    messages: [
      {
        role: 'system',
        content: "Answer the user's question accurately and concisely.",
      },
    ],
  },
});

Set ANTHROPIC_API_KEY in your .env file for this handler. If no package fits, write a custom handler. To learn how, read Use a custom handler.

Use a custom handler

A custom handler is an async function that calls your model and returns the output. Wrap it with create_handler() in Python or createHandler() in Node.js to declare the provider and mode it supports. The SDK calls it once for each row. Use a custom handler to call a provider the SDK has no package for, to call an internal gateway, or to replay stored outputs. Here is an example:

from typing import Any

from launchdarkly_ai_server import create_handler


async def handler(
    config: dict[str, Any],
    user_input: str | None = None,
    tool_handlers: dict[str, Any] | None = None,
    variables: dict[str, Any] | None = None,
) -> dict[str, Any]:
    return {"output": f"response for {user_input or ''}"}


handler = create_handler(("custom", "completion"), handler)

import { createHandler } from '@launchdarkly/ai-node';

const handler = createHandler(
  ['custom', 'messages'],
  async (
    config,
    userInput,
    toolHandlers,
    variables,
    history,
  ) => {
    return {
      output: `response for ${userInput ?? ''}`,
    };
  },
);

The first argument to create_handler() or createHandler() sets the provider and mode that the handler serves. The handler receives the following arguments:

Python
Node.js

config holds the provider, model, parameters, instructions or messages, and the tool definitions from generation.
user_input is the rendered row input.
tool_handlers maps each tool key to the function you passed in tools.
variables holds the row variables plus the rendered input and expected_output.

Return a dict with output. Add usage with input_tokens and output_tokens when you have them.

config holds the provider, model, parameters, instructions or messages, and the tool definitions from generation.
userInput is the rendered row input.
toolHandlers maps each tool key to the implementation or provider-native tool passed in tools.
variables holds the row variables plus the rendered input and expectedOutput.
history contains the message history when the handler is used in an agent or tool-calling flow.

Return an object with output. Add usage with input, output, and total token counts when the provider reports them.

LaunchDarkly records the usage on the row. To test a handler that calls no model, read Run your first evaluation in Python or Run your first evaluation in Node.js.

Install the SDK

The functions for running evaluations ship in the AI SDK package for your language: launchdarkly-ai-server for Python and @launchdarkly/ai-node for Node.js. You also install one handler package for each model provider you generate with. In Python, also install the base launchdarkly-server-sdk. The examples in this topic let init_evaluations() create the LaunchDarkly client that sends the results, and that client ships in the base SDK. You can omit it if your application already initializes a LaunchDarkly client and passes it to the SDK, or if the base SDK does not run on your platform. To learn more, read Bring your own LaunchDarkly client. In Node.js, @launchdarkly/ai-node includes the base Node.js SDK. Do not install the base Node.js SDK separately. Install the SDK and the handler package for the provider you chose in the previous section. For example, to install the SDK and a handler:

Python SDK
Node.js SDK

Shell (Python)

pip install launchdarkly-server-sdk launchdarkly-ai-server <YOUR-PROVIDER-PACKAGE> python-dotenv

Shell (Node.js)

npm install @launchdarkly/ai-node <YOUR-PROVIDER-PACKAGE> dotenv

How verdicts work

A criterion is a judge or scorer that grades each row. Each criterion has a threshold, which is the score a row must meet to pass that criterion. For a detailed explanation of how LaunchDarkly decides the verdicts, read How verdicts work.

AI SDKs

This feature is available for the following AI SDKs:

Python AI
Node.js (server-side) AI

Python AI

Expand Python AI SDK code samples

Run your first evaluation in Python

To run your first evaluation, complete the following steps in order.

Verify your setup in Python

Before you make any model calls, check that your credentials, dataset key, and event path work. The following script uses a custom handler that calls no model and returns a fixed output. It needs only your LaunchDarkly credentials. No provider key is required. Here is the script:

Python AI SDK

import asyncio
import os
import sys
from datetime import UTC, datetime
from typing import Any

from dotenv import load_dotenv
from launchdarkly_ai_server import create_handler, init_evaluations, shutdown


def required(name: str) -> str:
    value = os.getenv(name, "").strip()
    if not value:
        raise RuntimeError(f"Set {name} in the environment or .env")
    return value


async def deterministic_handler(
    config: dict[str, Any],
    user_input: str | None = None,
    tool_handlers: dict[str, Any] | None = None,
    variables: dict[str, Any] | None = None,
) -> dict[str, Any]:
    # Return a fixed output. A real handler calls the model here.
    return {"output": f"smoke response: {user_input or ''}"}


deterministic_handler = create_handler(
    ("custom", "completion"),
    deterministic_handler,
)


async def main() -> int:
    load_dotenv()
    required("LD_API_TOKEN")
    required("LD_SDK_KEY")
    project_key = required("LD_PROJECT_KEY")
    dataset_key = required("LD_DATASET_KEY")
    unique = datetime.now(UTC).strftime("%Y%m%d-%H%M%S-%f")

    evals = init_evaluations(project_key=project_key)
    try:
        result = await evals.run(
            key=f"verify-setup-{unique}",
            dataset=dataset_key,
            handler=deterministic_handler,
            generation={
                "provider": "custom",
                "model": "deterministic-smoke-handler",
                "mode": "completion",
                "instructions": "Return a deterministic smoke-test response.",
            },
            concurrency=2,
        )
        print(f"url={result.url}")
        print(f"summary={result.summary}")
        return 0 if result.passed else 1
    finally:
        await shutdown()


if __name__ == "__main__":
    sys.exit(asyncio.run(main()))

Save the script as verify_setup.py and run it:

Shell (Python)

python verify_setup.py

If the script prints a URL, your credentials, dataset key, and event path work. If it fails or times out, read Troubleshooting.

Run the evaluation in Python

The following script runs every row of the dataset through OpenAI and reports the results to LaunchDarkly. It uses no criteria, which makes it a generation-only run. Every row that produces an output passes. Here is the script:

Python AI SDK

import asyncio
import os
import sys
from datetime import UTC, datetime

from dotenv import load_dotenv
from launchdarkly_ai_openai_messages import create_openai_messages_handler
from launchdarkly_ai_server import init_evaluations, shutdown

MODEL = "gpt-4o"


def required(name: str) -> str:
    value = os.getenv(name, "").strip()
    if not value:
        raise RuntimeError(f"Set {name} in the environment or .env")
    return value


async def main() -> int:
    load_dotenv()
    required("LD_API_TOKEN")
    required("LD_SDK_KEY")
    required("OPENAI_API_KEY")
    project_key = required("LD_PROJECT_KEY")
    dataset_key = required("LD_DATASET_KEY")
    unique = datetime.now(UTC).strftime("%Y%m%d-%H%M%S-%f")

    evals = init_evaluations(project_key=project_key)
    try:
        result = await evals.run(
            key=f"first-eval-{unique}",  # Must be unique per run
            dataset=dataset_key,
            handler=create_openai_messages_handler(),
            generation={
                "provider": "OpenAI",
                "model": MODEL,
                "mode": "completion",
                "parameters": {"temperature": 0},
                "instructions": "Answer the user's question accurately and concisely.",
            },
            concurrency=5,
        )
        print(f"run_id={result.run_id}")
        print(f"url={result.url}")
        print(f"summary={result.summary}")
        print(f"stored_verdict={'passed' if result.passed else 'failed'}")
        return 0 if result.passed else 1
    finally:
        await shutdown()


if __name__ == "__main__":
    sys.exit(asyncio.run(main()))

Save the script as run_first_eval.py and run it:

Shell (Python)

python run_first_eval.py

If the run times out or no results appear in LaunchDarkly, read Troubleshooting.

Review your results in Python

The script prints four lines: the run ID, a URL, the summary, and whether the run passed. Here is an example:

Python AI SDK

run_id=22222222-2222-2222-2222-222222222222
url=https://app.launchdarkly.com/projects/my-project/ai/evaluations/11111111-1111-1111-1111-111111111111/runs/22222222-2222-2222-2222-222222222222
summary=RunSummary(total_rows=10, passed_rows=10, failed_rows=0, error_rows=0, pending_rows=0)
stored_verdict=passed

Open the URL to review the run in LaunchDarkly. You can also find the run from the Evaluations page. To learn how, read Review evaluation runs.The run_id and stored_verdict labels come from the script. The result object that the SDK returns has run_id, url, summary, and passed. The script prints result.summary directly. This means the summary uses the dataclass representation with snake_case field names, not JSON. The row counts vary by run.The terminal shows a short result that is useful in CI, and LaunchDarkly shows the detailed result. The process exits with code 0 when result.passed is true, and with 1 otherwise.The run URL has the following format:

Python AI SDK

{ui_base_uri}/projects/{project_key}/ai/evaluations/{evaluation_id}/runs/{run_id}

The default base URI is https://app.launchdarkly.com. To use another LaunchDarkly host, set LD_UI_BASE_URI, or pass ui_base_uri to init_evaluations(). The SDK URL-encodes the project key, evaluation ID, and run ID in the path. Here is an example:

Python AI SDK

evals = init_evaluations(
    project_key=project_key,
    ui_base_uri="https://app.launchdarkly.com",
)

Keep the following in mind about run():

Every call creates a new evaluation. Use a new key for each new run, such as one that includes a timestamp.
dataset is the dataset key from the Library, not its ID.
generation.mode is required when you provide generation directly. Use completion with a messages handler and agent with an agent handler.
generation.instructions is one system message, and generation.messages is a full message list. Do not pass both.
concurrency bounds how many rows run at once. The default is 10.
Large datasets might need poll_timeout_seconds above the default of 180.

Use an inline dataset in Python

An inline dataset is a test set that your script brings with it. Instead of passing a dataset key, you pass the dataset values directly from your code. To learn more, read Define datasets and tools in code.Inline dataset rows use camelCase field names, such as expectedOutput. Uploaded dataset files and DatasetRow use expected_output instead. Here is an example:

Python AI SDK

from launchdarkly_ai_openai_messages import create_openai_messages_handler

inline_dataset = [
    {
        "input": "What is the status of order {{order_id}}?",
        "expectedOutput": "Order A19 has shipped and is refund eligible.",
        "variables": {"order_id": "A19"},
        "metadata": {"expects_tool_call": True},
    },
]

result = await evals.run(
    key=f"inline-dataset-{unique}",
    dataset=inline_dataset,
    handler=create_openai_messages_handler(),
    generation={
        "provider": "OpenAI",
        "model": "gpt-4o",
        "mode": "completion",
        "instructions": "Answer accurately.",
    },
)

The inline dataset must be non-empty. Each row accepts only input, expectedOutput, variables, metadata, and rowIdx. The SDK rejects any other field. input must be a non-empty string. expectedOutput is optional and must be a string when supplied. variables and metadata must be JSON-serializable objects. rowIdx is optional and, when supplied, must match the row’s zero-based position.

Score rows in Python

You can score rows with scorers, LaunchDarkly judges, or both in the same run. To learn how scores become verdicts, read How verdicts work.A Scorer wraps a local function. The function receives the DatasetRow and the generated output. It returns a finite number from 0 to 1. Nothing else is accepted. A yes-or-no check returns 1.0 or 0.0. A row passes a scorer when its score meets the threshold.DatasetRow exposes row_index, input, expected_output, variables, and metadata. The rendered input and expected output are also available in variables. Here is an example of a scorer:

Python AI SDK

from launchdarkly_ai_server import DatasetRow, Scorer


def score_expected_text(row: DatasetRow, output: str | None) -> float:
    """Return 1.0 when the expected output appears in the generated output."""
    if not row.expected_output or not output:
        return 0.0
    expected = " ".join(row.expected_output.casefold().split())
    generated = " ".join(output.casefold().split())
    return 1.0 if expected in generated else 0.0


scorer = Scorer(
    name="expected-output-in-answer",
    fn=score_expected_text,
    threshold=1.0,
)

A Judge references a judge config by key. The SDK uses your SDK key to resolve the judge once per run, before it creates any records. A missing or disabled judge fails the run up front. Here is an example of a judge:

Python AI SDK

from launchdarkly_ai_server import Judge

JUDGE_KEY = "test-accuracy"

judge = Judge(key=JUDGE_KEY, threshold=0.8)

To use judges and scorers in the same run, pass both in criteria. The following example scores every row with a LaunchDarkly judge and a local scorer:

Python AI SDK

from launchdarkly_ai_openai_messages import create_openai_messages_handler
from launchdarkly_ai_server import Criterion, Judge, Scorer

MODEL = "gpt-4o"
JUDGE_KEY = "test-accuracy"


def score_nonempty_output(row: object, output: str | None) -> float:
    """Return 1.0 when the output has text."""
    return 1.0 if output and output.strip() else 0.0


criteria: list[Criterion] = [
    Judge(key=JUDGE_KEY),
    Scorer(name="nonempty-output", fn=score_nonempty_output, threshold=1.0),
]

result = await evals.run(
    key=f"judges-and-scorers-{unique}",
    dataset=dataset_key,
    handler=create_openai_messages_handler(),
    generation={
        "provider": "OpenAI",
        "model": MODEL,
        "mode": "completion",
        "parameters": {"temperature": 0},
        "instructions": "Answer the user's question accurately and concisely.",
    },
    criteria=criteria,
    concurrency=5,
)

Judge keys and scorer names share one namespace. They must be unique within a run, without regard to case.Because a judge is a model call, one of your handlers must serve the judge’s provider. If the judge uses a different provider than generation, pass both handlers in handlers. The SDK selects the handler whose provider and mode match the generation or judge. Pass exactly one of handler and handlers. To learn more, read Judges and scorers.For example, to generate with OpenAI and grade with Anthropic:

Python AI SDK

from launchdarkly_ai_claude_messages import create_claude_messages_handler
from launchdarkly_ai_openai_messages import create_openai_messages_handler
from launchdarkly_ai_server import Judge

JUDGE_KEY = "test-accuracy"

judge = Judge(key=JUDGE_KEY, threshold=0.8)

result = await evals.run(
    key=f"cross-provider-judges-{unique}",
    dataset=dataset_key,
    handlers=[
        create_openai_messages_handler(),
        create_claude_messages_handler(),
    ],
    generation={
        "provider": "OpenAI",
        "model": MODEL,
        "mode": "completion",
        "instructions": "Answer the user's question accurately and concisely.",
    },
    criteria=[judge],
)

The judge config determines which provider the judge uses. The handler pool supplies the matching handler.

Judge the tool trajectory in Python

A judge can grade the tool trajectory, which is how the agent reached its answer. The SDK records every tool call the row makes and renders it into {{message_history}} between the row input and the output. To learn more, read Tool trajectory.Scorers cannot see the trajectory, because a scorer receives only the row and the output. Use a judge for trajectory assertions.To judge the tool trajectory, complete the following steps in order.

Write the tool implementation in Python

The function receives the parsed arguments and returns a JSON-serializable value. Here is an example:

Python AI SDK

from collections.abc import Mapping
from typing import Any

_ORDERS = {
    "A19": {"status": "shipped", "refund_eligible": True},
    "B20": {"status": "processing", "refund_eligible": False},
}


def lookup_order(arguments: Mapping[str, Any]) -> dict[str, str | bool]:
    """Look up an order by its id."""
    order_id = arguments.get("order_id")
    if not isinstance(order_id, str) or not order_id.strip():
        raise ValueError("lookup_order requires a non-empty string order_id")
    order = _ORDERS.get(order_id)
    if order is None:
        return {"status": "not_found", "refund_eligible": False}
    return dict(order)

Make the tool available to the run in Python

Either create the tool in the Library, or define it inline.To use a Library tool, create a tool in the Library with the key lookup_order and the same schema as the inline example in this step. The argument names in the schema must match the arguments your implementation reads. In your script, resolve the tool with await evals.tools.get(), providing the tool key and its implementation. evals.tools.get() supplies the Library schema and description.To define the tool inline instead, construct an EvalTool with the key, the implementation, the schema, and the description. To learn more, read Define datasets and tools in code. Here is an example:

Python AI SDK

from launchdarkly_ai_server import EvalTool

inline_tool = EvalTool(
    key="lookup_order",
    implementation=lookup_order,
    schema={
        "type": "object",
        "properties": {"order_id": {"type": "string"}},
        "required": ["order_id"],
    },
    description="Look up an order by its ID.",
)

The tools argument receives the EvalTool:

Python AI SDK

result = await evals.run(
    key=f"inline-tool-{unique}",
    dataset=inline_dataset,
    handler=create_openai_messages_handler(),
    tools=[inline_tool],
    generation={
        "provider": "OpenAI",
        "model": "gpt-4o",
        "mode": "completion",
        "instructions": "Use lookup_order when an order ID is present.",
    },
)

Inline tool keys must be lowercase and unique within a run. The schema must be a JSON object. To move a tool into the Library, create the corresponding Library tool and change the code to use evals.tools.get(). The tool implementation remains in your application.

Build a dataset with rows that need the tool in Python

A trajectory judge can only grade rows that have a reason to call the tool. The example dataset has five rows:

Two rows that need one call each, for orders A19 and B20.
One row for an unknown order, to grade whether the agent reports the miss instead of inventing a status.
One row that needs two calls, to grade ordering and that both results reach the answer.
One control row that needs no tool, to keep the judge honest about unused tools.

Upload the dataset to the Library as a JSONL file. Here is an example dataset:

Python AI SDK (JSONL dataset file)

{"input": "What is the status of order {{order_id}}, and can I get a refund?", "expected_output": "Order A19 has shipped and is refund eligible.", "variables": {"order_id": "A19"}, "metadata": {"expects_tool_call": true}}
{"input": "Customer is asking about order {{order_id}}. Can they get their money back?", "expected_output": "Order B20 is still processing and is not refund eligible.", "variables": {"order_id": "B20"}, "metadata": {"expects_tool_call": true}}
{"input": "Look up order {{order_id}} and tell me whether it shipped.", "expected_output": "Order Z99 could not be found.", "variables": {"order_id": "Z99"}, "metadata": {"expects_tool_call": true}}
{"input": "Compare orders {{first_order}} and {{second_order}}: which one can be refunded?", "expected_output": "A19 is refund eligible; B20 is not.", "variables": {"first_order": "A19", "second_order": "B20"}, "metadata": {"expects_tool_call": true}}
{"input": "What is your refund policy in general?", "expected_output": "A general policy explanation with no order lookup.", "variables": {}, "metadata": {"expects_tool_call": false}}

Create the judge and write its rubric in Python

Create a judge in LaunchDarkly. Put only the grading criteria in the system prompt. Do not add {{message_history}} to the system prompt, because the judge config’s default assistant message already carries it. The SDK appends the JSON score format instructions. Do not restate them. Here is an example system prompt:

Judge system prompt

You are grading an assistant's TOOL USE, not its prose.

The message history shows the user's request, the tools the assistant
could call, the calls it actually made with their arguments and each
result or error, and its final answer.

Score 1.0 when the tool use was right for the request. That INCLUDES
calling no tool at all when the request did not need one: an available
tool left unused is not a mistake, and a request that needs no tool
must not be penalized for having none.

Score 0.0 when the assistant needed a tool and did not call it, called
the wrong tool, passed arguments the request does not support, ignored
what a tool returned, or repeated a failing call without adapting.

Score in between for partial credit.

Keep the clause about an unused tool. Without it, the judge fails every row whose correct behavior is to call nothing.

Run the trajectory evaluation in Python

Here is an example:

Python AI SDK

from launchdarkly_ai_openai_messages import create_openai_messages_handler
from launchdarkly_ai_server import Judge

MODEL = "gpt-4o"
TOOL_KEY = "lookup_order"
TRAJECTORY_JUDGE_KEY = "tool-trajectory-relevance"
JUDGE_THRESHOLD = 0.8

judge = Judge(key=TRAJECTORY_JUDGE_KEY, threshold=JUDGE_THRESHOLD)

library_tool = await evals.tools.get(
    TOOL_KEY,
    implementation=lookup_order,
)

result = await evals.run(
    key=f"tool-trajectory-judge-{unique}",
    dataset=dataset_key,
    handler=create_openai_messages_handler(),
    tools=[library_tool],
    generation={
        "provider": "OpenAI",
        "model": MODEL,
        "mode": "completion",
        "parameters": {"temperature": 0},
        "instructions": (
            f"Use the {TOOL_KEY} tool whenever an order identifier is "
            "present. Base the final answer on the tool result and answer "
            "concisely."
        ),
    },
    criteria=[judge],
    concurrency=2,
)

For a row that called one tool, {{message_history}} renders like this:

Python AI SDK

Use lookup_order for A19

Tools available: lookup_order
Tool calls made while producing the response, in order:
1. lookup_order
   arguments: {"order_id":"A19"}
   result: {"refund_eligible":true,"status":"shipped"}

Order A19 is shipped and refund eligible.

Keep the following in mind when you design a rubric:

Tools available lists what the model could call. A rubric can use it to grade tool selection.
A row with tools that called none of them reads No tool calls were made while producing the response.
A run with no tools adds no trajectory block. Do not point a trajectory judge at a dataset with no tool rows.
A failed call shows error: in place of result:. This is what makes recovery gradeable.
Provider built-in tools run inside the provider. They never appear in the trajectory.
The trajectory is capped per row. To learn about the limits, read Limits.

Node.js (server-side) AI

Expand Node.js (server-side) AI SDK code samples

Run your first evaluation in Node.js

To run your first evaluation, complete the following steps in order.

Verify your setup in Node.js

Node.js (server-side) AI SDK

import 'dotenv/config';
import { createHandler, initEvaluations, shutdown } from '@launchdarkly/ai-node';

const deterministicHandler = createHandler(
  ['custom', 'messages'],
  async (
    config,
    userInput,
    toolHandlers,
    variables,
    history,
  ) => ({
    output: `smoke response: ${userInput ?? ''}`,
  }),
);

const required = (name) => {
  const value = process.env[name]?.trim();
  if (!value) {
    throw new Error(`Set ${name} in the environment or .env`);
  }
  return value;
};

const projectKey = required('LD_PROJECT_KEY');
const datasetKey = required('LD_DATASET_KEY');
const evals = initEvaluations({ projectKey });
const unique = new Date().toISOString().replace(/[-:.TZ]/g, '');

try {
  const result = await evals.run({
    key: `verify-setup-${unique}`,
    dataset: datasetKey,
    handler: deterministicHandler,
    generation: {
      provider: 'custom',
      model: 'deterministic-smoke-handler',
      mode: 'completion',
      messages: [
        { role: 'system', content: 'Return a deterministic smoke-test response.' },
      ],
    },
    concurrency: 2,
  });

  console.log(`url=${result.url}`);
  console.log(`summary=${JSON.stringify(result.summary)}`);
  process.exitCode = result.passed ? 0 : 1;
} finally {
  await shutdown();
}

Save the script as verify_setup.mjs and run it:

Shell (Node.js)

node verify_setup.mjs

If the script prints a URL, your credentials, dataset key, and event path work. If it fails or times out, read Troubleshooting.

Run the evaluation in Node.js

Node.js (server-side) AI SDK

import 'dotenv/config';
import { createOpenAIHandler } from '@launchdarkly/ai-openai-messages';
import { initEvaluations, shutdown } from '@launchdarkly/ai-node';

const MODEL = 'gpt-4o';

const required = (name) => {
  const value = process.env[name]?.trim();
  if (!value) {
    throw new Error(`Set ${name} in the environment or .env`);
  }
  return value;
};

const projectKey = required('LD_PROJECT_KEY');
const datasetKey = required('LD_DATASET_KEY');
const evals = initEvaluations({ projectKey });
const unique = new Date().toISOString().replace(/[-:.TZ]/g, '');

try {
  const result = await evals.run({
    key: `first-eval-${unique}`,
    dataset: datasetKey,
    handler: createOpenAIHandler(),
    generation: {
      provider: 'OpenAI',
      model: MODEL,
      mode: 'completion',
      parameters: { temperature: 0 },
      messages: [
        {
          role: 'system',
          content: "Answer the user's question accurately and concisely.",
        },
      ],
    },
    concurrency: 5,
  });

  console.log(`run_id=${result.runId}`);
  console.log(`url=${result.url}`);
  console.log(`summary=${JSON.stringify(result.summary)}`);
  console.log(`stored_verdict=${result.passed ? 'passed' : 'failed'}`);
  process.exitCode = result.passed ? 0 : 1;
} finally {
  await shutdown();
}

Save the script as run_first_eval.mjs and run it:

Shell (Node.js)

node run_first_eval.mjs

If the run times out or no results appear in LaunchDarkly, read Troubleshooting.

Review your results in Node.js

The script prints four lines: the run ID, a URL, the summary, and whether the run passed. Here is an example:

Node.js (server-side) AI SDK

run_id=22222222-2222-2222-2222-222222222222
url=https://app.launchdarkly.com/projects/my-project/ai/evaluations/11111111-1111-1111-1111-111111111111/runs/22222222-2222-2222-2222-222222222222
summary={"totalRows":10,"passedRows":10,"failedRows":0,"errorRows":0,"pendingRows":0}
stored_verdict=passed

Open the URL to review the run in LaunchDarkly. You can also find the run from the Evaluations page. To learn how, read Review evaluation runs.The run_id and stored_verdict labels come from the script. The result object that the SDK returns has runId, url, summary, and passed. The script prints the summary as JSON with camelCase field names. The row counts vary by run.The terminal shows a short result that is useful in CI, and LaunchDarkly shows the detailed result. The process exits with code 0 when result.passed is true, and with 1 otherwise.The run URL has the following format:

Node.js (server-side) AI SDK

{uiBaseUri}/projects/{projectKey}/ai/evaluations/{evaluationId}/runs/{runId}

The default base URI is https://app.launchdarkly.com. To use another LaunchDarkly host, set LD_UI_BASE_URI, or pass uiBaseUri. The SDK URL-encodes the project key, evaluation ID, and run ID in the path.Keep the following in mind about run():

Every call creates a new evaluation. Use a new key for each new run, such as one that includes a timestamp.
dataset is the dataset key from the Library, not its ID.
generation.mode is required when you provide generation directly. Use completion with a messages handler and agent with an agent handler.
Use generation.messages with a messages handler and generation.instructions with an agent handler. Do not pass both.
concurrency bounds how many rows run at once. The default is 10.
Large datasets might need pollTimeoutSeconds above the default of 180.

Use an inline dataset in Node.js

Node.js (server-side) AI SDK

const inlineDataset = [
  {
    input: 'What is the status of order {{orderId}}?',
    expectedOutput: 'Order A19 has shipped and is refund eligible.',
    variables: { orderId: 'A19' },
    metadata: { expectsToolCall: true },
  },
];

const result = await evals.run({
  key: `inline-dataset-${unique}`,
  dataset: inlineDataset,
  handler: createOpenAIHandler(),
  generation: {
    provider: 'OpenAI',
    model: 'gpt-4o',
    mode: 'completion',
    messages: [{ role: 'system', content: 'Answer accurately.' }],
  },
});

The inline dataset must be non-empty. Each row accepts only input, expectedOutput, variables, metadata, and rowIdx. input must be a non-empty string. expectedOutput is optional and must be a string when supplied. variables and metadata must be JSON-serializable objects. rowIdx is optional and, when supplied, must match the row’s zero-based position.

Score rows in Node.js

You can score rows with scorers, LaunchDarkly judges, or both in the same run. To learn how scores become verdicts, read How verdicts work.A Scorer wraps a local function. The function receives the DatasetRow and the generated output. It returns a finite number from 0 to 1. Nothing else is accepted. A yes-or-no check returns 1.0 or 0.0. A row passes a scorer when its score meets the threshold.DatasetRow exposes rowIndex, input, expectedOutput, variables, and metadata. The rendered input and expected output are also available in variables. Here is an example of a scorer:

Node.js (server-side) AI SDK

import { Scorer } from '@launchdarkly/ai-node';

const scoreExpectedText = (row, output) => {
  if (!row.expectedOutput || !output) {
    return 0;
  }

  const expected = row.expectedOutput.trim().toLowerCase();
  const generated = output.trim().toLowerCase();
  return generated.includes(expected) ? 1 : 0;
};

const scorer = new Scorer(
  'expected-output-in-answer',
  scoreExpectedText,
  { threshold: 1.0 },
);

Node.js (server-side) AI SDK

import { Judge } from '@launchdarkly/ai-node';

const JUDGE_KEY = 'test-accuracy';

const judge = new Judge(JUDGE_KEY, {
  threshold: 0.8,
});

To use judges and scorers in the same run, pass both in criteria. The following example scores every row with a LaunchDarkly judge and a local scorer:

Node.js (server-side) AI SDK

import { createOpenAIHandler } from '@launchdarkly/ai-openai-messages';
import {
  Judge,
  Scorer,
} from '@launchdarkly/ai-node';

const MODEL = 'gpt-4o';
const JUDGE_KEY = 'test-accuracy';

const scoreNonemptyOutput = (_row, output) =>
  output && output.trim() ? 1 : 0;

const criteria = [
  new Judge(JUDGE_KEY),
  new Scorer('nonempty-output', scoreNonemptyOutput, { threshold: 1.0 }),
];

const result = await evals.run({
  key: `judges-and-scorers-${unique}`,
  dataset: datasetKey,
  handler: createOpenAIHandler(),
  generation: {
    provider: 'OpenAI',
    model: MODEL,
    mode: 'completion',
    parameters: { temperature: 0 },
    messages: [
      {
        role: 'system',
        content: "Answer the user's question accurately and concisely.",
      },
    ],
  },
  criteria,
  concurrency: 5,
});

Node.js (server-side) AI SDK

import { createClaudeMessagesHandler } from '@launchdarkly/ai-claude-messages';
import { createOpenAIHandler } from '@launchdarkly/ai-openai-messages';

const result = await evals.run({
  key: `cross-provider-judges-${unique}`,
  dataset: datasetKey,
  handlers: [
    createOpenAIHandler(),
    createClaudeMessagesHandler(),
  ],
  generation: {
    provider: 'OpenAI',
    model: 'gpt-4o',
    mode: 'completion',
    messages: [{ role: 'system', content: 'Answer accurately.' }],
  },
  criteria: [new Judge('anthropic-accuracy')],
});

The judge config determines which provider the judge uses. The handler pool supplies the matching handler.

Judge the tool trajectory in Node.js

Write the tool implementation in Node.js

The function receives the parsed arguments and returns a JSON-serializable value. Here is an example:

Node.js (server-side) AI SDK

const orders = {
  A19: { status: 'shipped', refundEligible: true },
  B20: { status: 'processing', refundEligible: false },
};

const lookupOrder = async ({ orderId }) => {
  if (typeof orderId !== 'string' || !orderId.trim()) {
    throw new Error('lookup_order requires a non-empty string orderId');
  }

  return orders[orderId] ?? {
    status: 'not_found',
    refundEligible: false,
  };
};

Make the tool available to the run in Node.js

Either create the tool in the Library, or define it inline.To use a Library tool, create a tool in the Library with the key lookup_order and the same schema as the inline example in this step. The argument names in the schema must match the arguments your implementation reads. In your script, resolve the tool with await evals.tools.get(), providing the tool key and its implementation. evals.tools.get() supplies the Library schema and description, and pins the tool version for the run.To define the tool inline instead, construct an EvalTool with the key, the implementation, the schema, and the description. To learn more, read Define datasets and tools in code. Here is an example:

Node.js (server-side) AI SDK

import { EvalTool } from '@launchdarkly/ai-node';

const inlineTool = new EvalTool(
  'lookup_order',
  lookupOrder,
  {
    type: 'object',
    properties: { orderId: { type: 'string' } },
    required: ['orderId'],
  },
  'Look up an order by its ID.',
);

The tools argument receives the EvalTool:

Node.js (server-side) AI SDK

const result = await evals.run({
  key: `inline-tool-${unique}`,
  dataset: inlineDataset,
  handler: createOpenAIHandler(),
  tools: [inlineTool],
  generation: {
    provider: 'OpenAI',
    model: 'gpt-4o',
    mode: 'completion',
    messages: [
      {
        role: 'system',
        content: 'Use lookup_order when an order ID is present.',
      },
    ],
  },
});

Inline tool keys must be lowercase and unique within a run. To move a tool into the Library, create the corresponding Library tool and change the code to use evals.tools.get(). The tool implementation remains in your application.

Build a dataset with rows that need the tool in Node.js

A trajectory judge can only grade rows that have a reason to call the tool. The example dataset has five rows:

Two rows that need one call each, for orders A19 and B20.
One row for an unknown order, to grade whether the agent reports the miss instead of inventing a status.
One row that needs two calls, to grade ordering and that both results reach the answer.
One control row that needs no tool, to keep the judge honest about unused tools.

Pass the rows as an inline dataset. Here is an example dataset:

Node.js (server-side) AI SDK

const trajectoryDataset = [
  {
    input: 'What is the status of order {{orderId}}, and can I get a refund?',
    expectedOutput: 'Order A19 has shipped and is refund eligible.',
    variables: { orderId: 'A19' },
    metadata: { expectsToolCall: true },
  },
  {
    input: 'Customer is asking about order {{orderId}}. Can they get their money back?',
    expectedOutput: 'Order B20 is still processing and is not refund eligible.',
    variables: { orderId: 'B20' },
    metadata: { expectsToolCall: true },
  },
  {
    input: 'Look up order {{orderId}} and tell me whether it shipped.',
    expectedOutput: 'Order Z99 could not be found.',
    variables: { orderId: 'Z99' },
    metadata: { expectsToolCall: true },
  },
  {
    input: 'Compare orders {{firstOrder}} and {{secondOrder}}: which one can be refunded?',
    expectedOutput: 'A19 is refund eligible; B20 is not.',
    variables: { firstOrder: 'A19', secondOrder: 'B20' },
    metadata: { expectsToolCall: true },
  },
  {
    input: 'What is your refund policy in general?',
    expectedOutput: 'A general policy explanation with no order lookup.',
    variables: {},
    metadata: { expectsToolCall: false },
  },
];

Create the judge and write its rubric in Node.js

Judge system prompt

You are grading an assistant's TOOL USE, not its prose.

The message history shows the user's request, the tools the assistant
could call, the calls it actually made with their arguments and each
result or error, and its final answer.

Score 1.0 when the tool use was right for the request. That INCLUDES
calling no tool at all when the request did not need one: an available
tool left unused is not a mistake, and a request that needs no tool
must not be penalized for having none.

Score 0.0 when the assistant needed a tool and did not call it, called
the wrong tool, passed arguments the request does not support, ignored
what a tool returned, or repeated a failing call without adapting.

Score in between for partial credit.

Keep the clause about an unused tool. Without it, the judge fails every row whose correct behavior is to call nothing.

Run the trajectory evaluation in Node.js

Here is an example:

Node.js (server-side) AI SDK

const trajectoryJudge = new Judge('tool-trajectory-relevance', {
  threshold: 0.8,
});

const libraryTool = await evals.tools.get('lookup_order', {
  implementation: lookupOrder,
});

const result = await evals.run({
  key: `tool-trajectory-judge-${unique}`,
  dataset: trajectoryDataset,
  handler: createOpenAIHandler(),
  tools: [libraryTool],
  generation: {
    provider: 'OpenAI',
    model: 'gpt-4o',
    mode: 'completion',
    messages: [
      {
        role: 'system',
        content:
          'Use lookup_order whenever an order identifier is present. ' +
          'Base the final answer on the tool result and answer concisely.',
      },
    ],
  },
  criteria: [trajectoryJudge],
  concurrency: 2,
});

For a row that called one tool, {{message_history}} renders like this:

Node.js (server-side) AI SDK

Use lookup_order for A19

Tools available: lookup_order
Tool calls made while producing the response, in order:
1. lookup_order
   arguments: {"orderId":"A19"}
   result: {"refundEligible":true,"status":"shipped"}

Order A19 is shipped and refund eligible.

Keep the following in mind when you design a rubric:

Tools available lists what the model could call. A rubric can use it to grade tool selection.
A row with tools that called none of them reads No tool calls were made while producing the response.
A run with no tools adds no trajectory block. Do not point a trajectory judge at a dataset with no tool rows.
A failed call shows error: in place of result:. This is what makes recovery gradeable.
Provider built-in tools run inside the provider. They never appear in the trajectory.
The trajectory is capped per row. To learn about the limits, read Limits.

Run evaluations in CI

The script in Run your first evaluation in Python or Run your first evaluation in Node.js exits with code 0 when the stored run passes, and with 1 when it does not. Most CI systems treat a non-zero exit code as a failed step. This means the result of a run can fail a build or block a release. Keep the following in mind when you run in CI:

Store credentials as secrets: Add LD_API_TOKEN and LD_SDK_KEY, the project key, and any provider key as secrets in your CI system. Pass them to the job as environment variables. Do not commit them.
Use a unique run key for each new run: Every run creates a new evaluation. A timestamp works, and you can also include the CI run number.
Retry a failed run with the same key: To retry a run that failed, run the script again with the same run key.
Create Library assets first: Library datasets, tools, and judges must exist in the project before the job runs. Inline datasets and inline tools live in your code and travel with your repository.
Raise limits for large datasets: Large datasets might need poll_timeout_seconds in Python, or pollTimeoutSeconds in Node.js, above the default of 180.
Open the run URL for details: The terminal shows a short result, and LaunchDarkly shows the detailed result.

Options

This section lists the arguments and settings that control a run.

Run options

Pass the key of the LaunchDarkly project that holds the dataset, tools, and judges when you initialize evaluations. In Python, pass project_key to init_evaluations(). In Node.js, pass projectKey to initEvaluations(), or set LD_PROJECT_KEY. The run() function accepts the following arguments:

Python
Node.js

Argument	Description
`key`	(Required) The key for the evaluation. Use a new key for each new run. To retry a failed run, reuse its key.
`dataset`	(Required) The dataset key from the Library, not its ID, or a list of inline dataset rows.
`handler`	(Required unless you pass `handlers`) A LaunchDarkly handler or a custom handler function.
`generation`	(Required) The `provider`, `model`, `mode`, `parameters`, and either `instructions` or `messages`. `mode` is required when you provide `generation` directly. Use `instructions` for one system message and `messages` for a full message list. Do not pass both.
`tools`	A list of `EvalTool` objects. Use `evals.tools.get()` for Library tools or construct an `EvalTool` for an inline tool.
`criteria`	A list of `Judge` and `Scorer` objects. With no criteria, every row that produces an output passes.
`handlers`	A list of handlers used when generation and judges require different provider or mode combinations. Pass exactly one of `handler` and `handlers`.
`concurrency`	How many rows run at once. Defaults to 10.
`poll_interval_seconds`	How many seconds the SDK waits between checks of the run summary. Defaults to 2 seconds.
`poll_timeout_seconds`	How long the SDK waits for the run to finish. Large datasets might need a higher value. Defaults to 180 seconds.

Argument	Description
`key`	(Required) The key for the evaluation. Use a new key for each new run. To retry a failed run, reuse its key.
`dataset`	(Required) The dataset key from the Library, not its ID, or an array of inline dataset rows.
`handler`	(Required unless you pass `handlers`) A LaunchDarkly handler, or a custom handler built with `createHandler()`.
`generation`	(Required) The `provider`, `model`, `mode`, `parameters`, and either `messages` or `instructions`. `mode` is `completion` or `agent`, and is required when you provide `generation` directly. Use `messages` with a messages handler and `instructions` with an agent handler. Do not pass both.
`tools`	An array of `EvalTool` instances. Use `evals.tools.get()` for Library tools or construct an `EvalTool` for an inline tool.
`criteria`	An array of `Judge` and `Scorer` instances. With no criteria, every row that produces an output passes.
`handlers`	An array of handlers used when generation and judges require different provider or mode combinations. Pass exactly one of `handler` and `handlers`.
`concurrency`	How many rows run at once. Defaults to 10.
`pollIntervalSeconds`	How many seconds the SDK waits between checks of the run summary. Defaults to 2 seconds.
`pollTimeoutSeconds`	How long the SDK waits for the run to finish. Large datasets might need a higher value. Defaults to 180 seconds.

Scorer settings

A Scorer accepts the following settings:

Python
Node.js

Setting	Description
`name`	(Required) The name of the scorer. Names and judge keys share one namespace and must be unique within a run, without regard to case.
`fn`	(Required) The function that receives the `DatasetRow` and the output, and returns a number from 0 to 1.
`threshold`	The score a row must meet to pass. Defaults to 1.0, which means a row passes only on a perfect score. Lower it for graded scorers.
`success_direction`	`"higher_is_better"` or `"lower_is_better"`. Use `"lower_is_better"` for a scorer that counts something unwanted. Defaults to `"higher_is_better"`.

The Scorer constructor takes the name and the function, followed by an options object with the remaining settings.

Setting	Description
`name`	(Required) The name of the scorer. Names and judge keys share one namespace and must be unique within a run, without regard to case.
`fn`	(Required) The function that receives the `DatasetRow` and the output, and returns a number from 0 to 1.
`threshold`	The score a row must meet to pass. Defaults to 1.0, which means a row passes only on a perfect score. Lower it for graded scorers.
`successDirection`	`'higher_is_better'` or `'lower_is_better'`. Use `'lower_is_better'` for a scorer that counts something unwanted. Defaults to `'higher_is_better'`.

Judge settings

A Judge accepts the following settings:

Python
Node.js

Setting	Description
`key`	(Required) The key of the judge config.
`threshold`	The score a row must meet to pass. A row passes when the judge score meets it. Defaults to 0.5.
`ground_truth_context`	Replaces the expected output as the reference the judge grades against.

The Judge constructor takes the key, followed by an options object with the remaining settings.

Setting	Description
`key`	(Required) The key of the judge config.
`threshold`	The score a row must meet to pass. A row passes when the judge score meets it. Defaults to 0.5.
`groundTruthContext`	Replaces the expected output as the reference the judge grades against.

The success direction of a judge lives on the judge config in LaunchDarkly. You do not set it in code.

Limits

The following limits apply to evaluation runs that you start from code:

Limit	Value	What happens when exceeded
Uploaded dataset file size	Under 32 MB	The upload fails validation and LaunchDarkly shows an error.
Rows in an uploaded dataset	Fewer than 10,000	The upload fails validation and LaunchDarkly shows an error.

The SDK records up to 50 tool calls per row, and up to 2,000 characters for each set of tool arguments or each result. For the default concurrency, poll_timeout_seconds, and pollTimeoutSeconds values, read Run options.

Troubleshooting

This table includes possible errors and their solutions:

Python
Node.js

Symptom	Likely cause	Resolution
`ImportError` for `EvalTool`, `create_handler`, or another name this guide imports	The installed SDK is too old.	Install `launchdarkly-ai-server` 1.0.0 or later.
`EvaluationsError: No LaunchDarkly SDK key provided`	`LD_SDK_KEY` is not set. Every path needs it.	Set `LD_SDK_KEY` in your environment or `.env` file, then run the script again.
The run times out and no results appear	The SDK key is missing or wrong, which means LaunchDarkly ingests no results.	Set `LD_SDK_KEY` to the server-side SDK key for the environment you want results in.
`HTTP 401`	The API access token is missing or invalid.	Check `LD_API_TOKEN`. Create a new token if the old one expired.
`HTTP 403`	The token lacks a dataset, tool, evaluation, or run permission.	Use the Writer role, or update the custom role to allow reading datasets and tools and creating evaluations and evaluation runs in the target project.
`HTTP 404`	The project key, dataset key, tool key, or judge key does not exist in the project you passed.	Check that all of the keys belong to the same project.
The run fails before it creates any records, and names a tool key	A tool key you passed in `tools` is not in the Library and was not defined inline.	Create the tool in the Library with the same key, or define it inline.
Judge resolution fails before the run starts	The SDK key is wrong, the judge config does not exist or has been disabled, or your handler cannot serve the judge’s provider.	Check `LD_SDK_KEY`, confirm the judge config exists and is enabled, and confirm your handler can serve the judge’s provider. If the judge uses a different provider, include handlers for both providers in `handlers`.
The run stays pending	Not every row was accepted, or events did not flush.	Confirm that every row was accepted and that your script calls `shutdown()` so events flush. Raise `poll_timeout_seconds` for large datasets.
The trajectory judge fails rows that need no tool	The rubric penalizes an unused tool.	Add the unused-tool clause to the rubric, or point the judge at a dataset where every row needs the tool.

Symptom	Likely cause	Resolution
An import of `initEvaluations`, `EvalTool`, or another name this guide imports fails	The installed SDK is too old.	Install `@launchdarkly/ai-node` 1.0.0 or later.
`EvaluationsError` that names `LD_PROJECT_KEY`, `LD_API_TOKEN`, or `LD_SDK_KEY`	That value is not set. `initEvaluations()` names the first missing value.	Set the value in your environment or `.env` file, or pass it to `initEvaluations()`, then run the script again.
The run times out and no results appear	The SDK key is missing or wrong, which means LaunchDarkly ingests no results.	Set `LD_SDK_KEY` to the server-side SDK key for the environment you want results in.
`HTTP 401`	The API access token is missing or invalid.	Check `LD_API_TOKEN`. Create a new token if the old one expired.
`HTTP 403`	The token lacks a dataset, tool, evaluation, or run permission.	Use the Writer role, or update the custom role to allow reading datasets and tools and creating evaluations and evaluation runs in the target project.
`HTTP 404`	The project key, dataset key, tool key, or judge key does not exist in the project you passed.	Check that all of the keys belong to the same project.
`generation.mode is required. Set it to 'completion' or 'agent'.`	`generation` has no `mode`.	Set `mode` to `completion` for a messages handler, or `agent` for an agent handler.
The run fails because the generation needs `messages`, not `instructions`, or the reverse	The prompt field does not match the handler’s mode.	Use `messages` with a messages handler and `instructions` with an agent handler.
The run fails because a handler does not declare `providesFor`	A custom handler is a plain function.	Wrap the handler with `createHandler()` to declare the provider and mode that it serves.
The run fails before it creates any records, and names a tool key	A tool key you passed in `tools` is not in the Library and was not defined inline.	Create the tool in the Library with the same key, or define it inline.
Judge resolution fails before the run starts	The SDK key is wrong, the judge config does not exist or has been disabled, or your handler cannot serve the judge’s provider.	Check `LD_SDK_KEY`, confirm the judge config exists and is active, and confirm your handler can serve the judge’s provider. If the judge uses a different provider, include handlers for both providers in `handlers`.
The run stays pending	Not every row was accepted, or events did not flush.	Confirm that every row was accepted and that your script calls `shutdown()` so events flush. Raise `pollTimeoutSeconds` for large datasets.
The trajectory judge fails rows that need no tool	The rubric penalizes an unused tool.	Add the unused-tool clause to the rubric, or point the judge at a dataset where every row needs the tool.

​How evaluations from code work

​Prerequisites

​Get your credentials

​Generate your credentials

​Write the .env file

​Add your provider credentials

​Choose a provider handler

​Use a custom handler

​Install the SDK

​How verdicts work

​AI SDKs

​Python AI

​Run your first evaluation in Python

Verify your setup in Python

Run the evaluation in Python

Review your results in Python

​Use an inline dataset in Python

​Score rows in Python

​Judge the tool trajectory in Python

Write the tool implementation in Python

Make the tool available to the run in Python

Build a dataset with rows that need the tool in Python

Create the judge and write its rubric in Python

Run the trajectory evaluation in Python

​Node.js (server-side) AI

​Run your first evaluation in Node.js

Verify your setup in Node.js

Run the evaluation in Node.js

Review your results in Node.js

​Use an inline dataset in Node.js

​Score rows in Node.js

​Judge the tool trajectory in Node.js

Write the tool implementation in Node.js

Make the tool available to the run in Node.js

Build a dataset with rows that need the tool in Node.js

Create the judge and write its rubric in Node.js

Run the trajectory evaluation in Node.js

​Run evaluations in CI

​Options

​Run options

​Scorer settings

​Judge settings

​Limits

​Troubleshooting

How evaluations from code work

Prerequisites

Get your credentials

Generate your credentials

Write the .env file

Add your provider credentials

Choose a provider handler

Use a custom handler

Install the SDK

How verdicts work

AI SDKs

Python AI

Run your first evaluation in Python

Use an inline dataset in Python

Score rows in Python

Judge the tool trajectory in Python

Node.js (server-side) AI

Run your first evaluation in Node.js

Use an inline dataset in Node.js

Score rows in Node.js

Judge the tool trajectory in Node.js

Run evaluations in CI

Options

Run options

Scorer settings

Judge settings

Limits

Troubleshooting