> ## Documentation Index
> Fetch the complete documentation index at: https://launchdarkly.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Optimization runs without a config

<View title="Developer" />

<View title="Federal docs" />

<View title="EU docs" />

This topic explains how to define judges, model choices, and evaluation parameters directly in code without a LaunchDarkly configuration object. This lets you perform optimization runs without an optimization config.

## Code example

Here is a complete example using `optimize_from_options` with the Claude Agent SDK:

<CodeGroup>
  ```python title="Python" expandable lines wrap theme={null}
  # LD imports
  from ldai_optimization import (
      OptimizationClient,
      OptimizationOptions,
      OptimizationJudge,
      OptimizationResponse,
      LLMCallConfig,
      LLMCallContext,
  )
  from ldai import LDAIClient
  from ldai.tracker import TokenUsage
  from ldclient import Context

  # anthropic imports
  from claude_agent_sdk import query, ClaudeAgentOptions
  from claude_agent_sdk.types import ResultMessage

  acceptance_statement_prompt = """\
  The orchestrator should appropriately fetch the user preferences and \
  route to the correct sub-agent, \
  carrying through any relevant information from the users' query.
  The orchestrator should not provide any answers itself, \
  just pass to the correct sub-agent.
  Inability to fetch user preferences or mentions of missing data should \
  be automatic failures.
  If preferences are not included, that should be an automatic failure.
  If the orchestrator does not mention the sub-agent it will hand off to, \
  that is an automatic failure."""

  default_fallback_model = "claude-opus-4-5-20251101"

  async def handle_agent_call(
      key: str,
      config: LLMCallConfig,
      context: LLMCallContext,
      is_evaluation: bool = False,
  ) -> OptimizationResponse:
      model = config.model.name if config.model else default_fallback_model
      final_message = None
      async for message in query(
          prompt=context.user_input or "",
          options=ClaudeAgentOptions(
              system_prompt=config.instructions or "",
              model=model,
          ),
      ):
          final_message = message

      if not isinstance(final_message, ResultMessage):
          raise ValueError(f"Unexpected final message type: {type(final_message)}")

      u = final_message.usage or {}
      input_tokens = u.get("input_tokens", 0)
      output_tokens = u.get("output_tokens", 0)

      return OptimizationResponse(
          output=final_message.result or "",
          usage=TokenUsage(
              total=input_tokens + output_tokens,
              input=input_tokens,
              output=output_tokens,
          ),
      )

  options = OptimizationOptions(
      judges={
          "acceptance": OptimizationJudge(
              acceptance_statement=acceptance_statement_prompt,
              threshold=0.95,
          ),
          "accuracy": OptimizationJudge(
              judge_key="my-accuracy-judge",
              threshold=0.8,
          ),
      },
      context_choices=[
          Context.builder("user-123").set("user_id", "user-123").build(),
      ],
      max_attempts=25,
      model_choices=["claude-opus-4-5", "claude-sonnet-4-5", "claude-haiku-4-5"],
      judge_model="claude-haiku-4-5",
      variable_choices=[
          {
              "user_id": "user-123",
              "trip_purpose": "business",
          },
          {
              "user_id": "user-125",
              "trip_purpose": "personal",
          },
      ],
      user_input_options=[
          "I'm going to tokyo next week, where should I stay near Shinjuku?",
          "where to eat in anchorage",
          "airbnbs near tahoe",
          "what are some food options in sf near the airport"
      ],
      handle_agent_call=handle_agent_call,
      handle_judge_call=handle_agent_call,
  )

  client = OptimizationClient(ld_ai_client)
  result = await client.optimize_from_options("travel-agent-orchestrator", options)
  ```
</CodeGroup>

## Code explanation

### Section 1: Handlers

Section 1 sets up the agent call for our provider to handle the LLM invocations. These methods are intended to be provider-agnostic, so you can use your own models as long as your orchestrator or framework can reach them.

There are two different handlers. They are:

* `handle_agent_call` executes the actual agent and receives a response. This code path is also used to generate new variations when the system requires it. To allow this, pass the calls through as generically as possible. Evaluators may need access to your tools for evaluation, so make sure it's configured for both execution and new variation generation.
* `handle_judge_call` (optional) executes the evaluation calls. It falls back to `handle_agent_call` if not provided. Specify this as a discrete handler if you want to capture response information, log metrics, or otherwise run judge calls differently from agent calls.

Your evaluator may need access to the same tools as your model. If you separate evaluator and model calls, specify tools for both.

### Section 2: Optimization parameters

Section 2 sets up the optimization parameters. Here's how:

* `judges` defines the scoring mechanisms used to find prompt acceptance. Judges can be acceptance statements or judge configs. To learn more, read [Performing optimization runs](/docs/home/agentcontrol/optimization-runs).
* `context_choices` (optional) defines the list of LaunchDarkly contexts that can be automatically chosen from when running a completion.
* `max_attempts` sets the maximum number of iterations allowed before failing. Use this to prevent over-spending on complex optimizations that may not reach a useful outcome. This does not correlate to the exact number of LLM calls made. Each optimization attempt incurs multiple LLM calls.
* `model_choices` defines the models that the optimizer is allowed to choose between throughout the optimization process.
* `judge_model` defines the model used for judging. This remains consistent across executions unless you intervene.
* `variable_choices` defines a list of variable choice sets that are randomly chosen when executing the agent. Larger numbers of `variable_choices` and `user_input_options` leads to more possible permutations and lessens the chance of instruction overfitting.
* `user_input_options` defines a list of user inputs that are randomly chosen when executing the agent.
* `handle_agent_call` executes the agent calls.
* `handle_judge_call` (optional) executes the evaluation (judge) calls. It falls back to `handle_agent_call` if not provided.

### Output

In (3), we're initializing the client and running the actual optimization. If you have info logging turned on, you can read the process as it runs in your logs. Here's an example of the output with info logging turned on:

<CodeGroup>
  ```bash title="Bash" lines wrap theme={null}
  INFO:ldai_optimization.client:[Iteration 1] -> Starting (attempt 1/10, model=claude-haiku-4-5)
  INFO:ldai_optimization.client:[Iteration 1] -> Calling agent (model=claude-haiku-4-5)...
  INFO:ldai_optimization.client:[Iteration 1] -> Executing evaluation...
  INFO:ldai_optimization.client:[Iteration 1] -> Running judge 1/2 'acceptance-statement-0' (acceptance)...
  INFO:ldai_optimization.client:[Iteration 1] -> Running judge 2/2 'ld-ai-judge-accuracy-1761841832389' (config)...
  INFO:ldai_optimization.client:[Iteration 1] -> One or more judges failed (attempt 1/10) — generating new variation
  INFO:ldai_optimization.client:[Iteration 1] -> Generating new variation...
  INFO:ldai_optimization.client:[Iteration 1] -> Model updated from 'claude-haiku-4-5' to 'claude-sonnet-4-5'
  INFO:ldai_optimization.client:[Iteration 2] -> Starting (attempt 2/10, model=claude-sonnet-4-5)
  INFO:ldai_optimization.client:[Iteration 2] -> Calling agent (model=claude-sonnet-4-5)...
  INFO:ldai_optimization.client:[Iteration 2] -> Executing evaluation...
  INFO:ldai_optimization.client:[Iteration 2] -> Running judge 1/2 'acceptance-statement-0' (acceptance)...
  INFO:ldai_optimization.client:[Iteration 2] -> Running judge 2/2 'ld-ai-judge-accuracy-1761841832389' (config)...
  INFO:ldai_optimization.client:[Iteration 2] -> One or more judges failed (attempt 2/10) — generating new variation
  INFO:ldai_optimization.client:[Iteration 2] -> Generating new variation...
  ```
</CodeGroup>

This call returns an `OptimizationContext` representing the final output of the optimization. You can access `OptimizationContext.history` to get all of the historical `OptimizationContext`s generated during the run.

Here is an abbreviated example output:

<CodeGroup>
  ```json title="JSON" expandable lines wrap theme={null}
  {
    "scores": {
      "acceptance-statement-0": {
        "score": 1.0,
        "rationale": "The response cleanly routes to the lodging-agent...",
        "duration_ms": 9795.460292021744,
        "usage": {
          "total": 4050,
          "input": 3088,
          "output": 962
        }
      }
    },
    "completion_response": "Routing to **lodging-agent**.\n\nHandoff context:\n- User ID placeholder: **user-125**...",
    "current_parameters": {
      // ... model parameters used for this iteration ...
    },
    "current_variables": {
      "trip_purpose": "business",
      "user_id": "user-125"
    },
    "current_model": "claude-sonnet-4-5",
    "user_input": "airbnbs near tahoe",
    "history": [
      // ... optimization context history ...
    ],
    "iteration": 5,
    "duration_ms": 2536.8339580018073,
    "usage": {
      "total": 1234,
      "input": 1109,
      "output": 125
    }
  }
  ```
</CodeGroup>