Optimization runs without a config | LaunchDarkly

This topic explains how to define judges, model choices, and evaluation parameters directly in code without a LaunchDarkly configuration object. This lets you perform optimization runs without an optimization config.

Code example

Here is a complete example using optimize_from_options with the Claude Agent SDK:

Python

# LD imports
from ldai_optimization import (
    OptimizationClient,
    OptimizationOptions,
    OptimizationJudge,
    OptimizationResponse,
    LLMCallConfig,
    LLMCallContext,
)
from ldai import LDAIClient
from ldai.tracker import TokenUsage
from ldclient import Context
# anthropic imports
from claude_agent_sdk import query, ClaudeAgentOptions
from claude_agent_sdk.types import ResultMessage
acceptance_statement_prompt = """\
The orchestrator should appropriately fetch the user preferences and \
route to the correct sub-agent, \
carrying through any relevant information from the users' query.
The orchestrator should not provide any answers itself, \
just pass to the correct sub-agent.
Inability to fetch user preferences or mentions of missing data should \
be automatic failures.
If preferences are not included, that should be an automatic failure.
If the orchestrator does not mention the sub-agent it will hand off to, \
that is an automatic failure."""
default_fallback_model = "claude-opus-4-5-20251101"
async def handle_agent_call(
    key: str,
    config: LLMCallConfig,
    context: LLMCallContext,
    is_evaluation: bool = False,
) -> OptimizationResponse:
    model = config.model.name if config.model else default_fallback_model
    final_message = None
    async for message in query(
        prompt=context.user_input or "",
        options=ClaudeAgentOptions(
            system_prompt=config.instructions or "",
            model=model,
        ),
    ):
        final_message = message
    if not isinstance(final_message, ResultMessage):
        raise ValueError(f"Unexpected final message type: {type(final_message)}")
    u = final_message.usage or {}
    input_tokens = u.get("input_tokens", 0)
    output_tokens = u.get("output_tokens", 0)
    return OptimizationResponse(
        output=final_message.result or "",
        usage=TokenUsage(
            total=input_tokens + output_tokens,
            input=input_tokens,
            output=output_tokens,
        ),
    )
options = OptimizationOptions(
    judges={
        "acceptance": OptimizationJudge(
            acceptance_statement=acceptance_statement_prompt,
            threshold=0.95,
        ),
        "accuracy": OptimizationJudge(
            judge_key="my-accuracy-judge",
            threshold=0.8,
        ),
    },
    context_choices=[
        Context.builder("user-123").set("user_id", "user-123").build(),
    ],
    max_attempts=25,
    model_choices=["claude-opus-4-5", "claude-sonnet-4-5", "claude-haiku-4-5"],
    judge_model="claude-haiku-4-5",
    variable_choices=[
        {
            "user_id": "user-123",
            "trip_purpose": "business",
        },
        {
            "user_id": "user-125",
            "trip_purpose": "personal",
        },
    ],
    user_input_options=[
        "I'm going to tokyo next week, where should I stay near Shinjuku?",
        "where to eat in anchorage",
        "airbnbs near tahoe",
        "what are some food options in sf near the airport"
    ],
    handle_agent_call=handle_agent_call,
    handle_judge_call=handle_agent_call,
)
client = OptimizationClient(ld_ai_client)
result = await client.optimize_from_options("travel-agent-orchestrator", options)

Code explanation

Section 1: Handlers

Section 1 sets up the agent call for our provider to handle the LLM invocations. These methods are intended to be provider-agnostic, so you can use your own models as long as your orchestrator or framework can reach them.

There are two different handlers. They are:

handle_agent_call executes the actual agent and receives a response. This code path is also used to generate new variations when the system requires it. To allow this, pass the calls through as generically as possible. Evaluators may need access to your tools for evaluation, so make sure it’s configured for both execution and new variation generation.
handle_judge_call (optional) executes the evaluation calls. It falls back to handle_agent_call if not provided. Specify this as a discrete handler if you want to capture response information, log metrics, or otherwise run judge calls differently from agent calls.

Your evaluator may need access to the same tools as your model. If you separate evaluator and model calls, specify tools for both.

Section 2: Optimization parameters

Section 2 sets up the optimization parameters. Here’s how:

judges defines the scoring mechanisms used to find prompt acceptance. Judges can be acceptance statements or judge configs. To learn more, read Performing optimization runs.
context_choices (optional) defines the list of LaunchDarkly contexts that can be automatically chosen from when running a completion.
max_attempts sets the maximum number of iterations allowed before failing. Use this to prevent over-spending on complex optimizations that may not reach a useful outcome. This does not correlate to the exact number of LLM calls made. Each optimization attempt incurs multiple LLM calls.
model_choices defines the models that the optimizer is allowed to choose between throughout the optimization process.
judge_model defines the model used for judging. This remains consistent across executions unless you intervene.
variable_choices defines a list of variable choice sets that are randomly chosen when executing the agent. Larger numbers of variable_choices and user_input_options leads to more possible permutations and lessens the chance of instruction overfitting.
user_input_options defines a list of user inputs that are randomly chosen when executing the agent.
handle_agent_call executes the agent calls.
handle_judge_call (optional) executes the evaluation (judge) calls. It falls back to handle_agent_call if not provided.

Output

In (3), we’re initializing the client and running the actual optimization. If you have info logging turned on, you can read the process as it runs in your logs. Here’s an example of the output with info logging turned on:

Bash

INFO:ldai_optimization.client:[Iteration 1] -> Starting (attempt 1/10, model=claude-haiku-4-5)
INFO:ldai_optimization.client:[Iteration 1] -> Calling agent (model=claude-haiku-4-5)...
INFO:ldai_optimization.client:[Iteration 1] -> Executing evaluation...
INFO:ldai_optimization.client:[Iteration 1] -> Running judge 1/2 'acceptance-statement-0' (acceptance)...
INFO:ldai_optimization.client:[Iteration 1] -> Running judge 2/2 'ld-ai-judge-accuracy-1761841832389' (config)...
INFO:ldai_optimization.client:[Iteration 1] -> One or more judges failed (attempt 1/10) — generating new variation
INFO:ldai_optimization.client:[Iteration 1] -> Generating new variation...
INFO:ldai_optimization.client:[Iteration 1] -> Model updated from 'claude-haiku-4-5' to 'claude-sonnet-4-5'
INFO:ldai_optimization.client:[Iteration 2] -> Starting (attempt 2/10, model=claude-sonnet-4-5)
INFO:ldai_optimization.client:[Iteration 2] -> Calling agent (model=claude-sonnet-4-5)...
INFO:ldai_optimization.client:[Iteration 2] -> Executing evaluation...
INFO:ldai_optimization.client:[Iteration 2] -> Running judge 1/2 'acceptance-statement-0' (acceptance)...
INFO:ldai_optimization.client:[Iteration 2] -> Running judge 2/2 'ld-ai-judge-accuracy-1761841832389' (config)...
INFO:ldai_optimization.client:[Iteration 2] -> One or more judges failed (attempt 2/10) — generating new variation
INFO:ldai_optimization.client:[Iteration 2] -> Generating new variation...

This call returns an OptimizationContext representing the final output of the optimization. You can access OptimizationContext.history to get all of the historical OptimizationContexts generated during the run.

Here is an abbreviated example output:

JSON

{
  "scores": {
    "acceptance-statement-0": {
      "score": 1.0,
      "rationale": "The response cleanly routes to the lodging-agent...",
      "duration_ms": 9795.460292021744,
      "usage": {
        "total": 4050,
        "input": 3088,
        "output": 962
      }
    }
  },
  "completion_response": "Routing to **lodging-agent**.\n\nHandoff context:\n- User ID placeholder: **user-125**...",
  "current_parameters": {
    // ... model parameters used for this iteration ...
  },
  "current_variables": {
    "trip_purpose": "business",
    "user_id": "user-125"
  },
  "current_model": "claude-sonnet-4-5",
  "user_input": "airbnbs near tahoe",
  "history": [
    // ... optimization context history ...
  ],
  "iteration": 5,
  "duration_ms": 2536.8339580018073,
  "usage": {
    "total": 1234,
    "input": 1109,
    "output": 125
  }
}