Expected output mode

This topic explains how to use Expected output mode to optimize an agent against known input/output pairs. This lets you optimize an agent against its expected outputs.

These optimizations compare the output of a given invocation against both the acceptance criteria and judges as well as the provided output to ensure it conforms to expectations.

How Expected output mode works

When you run an optimization on ground truth responses, it’s important to ensure that the optimization result takes into account and scores all expected input/variable/output pairs.

To accomplish this, Expected output mode does the following things:

The optimization scores each triplet of input, output, and variables against each of the judges.
The system collects judge responses. If any results fail, it generates a new candidate variation or agent.
The system scores the new candidate against each of the inputs again.

This process repeats until either:

A successful candidate passes all input triplets.
Or the run reaches the maximum number of attempts.

After a candidate succeeds, the system returns the candidate variation or agent. If you use the optimize_from_config method or auto_commit is true for the *_from_options methods, the system pushes that candidate to the agent config as a new variation.

Expected output mode can use more resources.

Because it mode requires complete executions of all available tests and may require multiple runs to develop a variation that passes, Expected output mode has a much higher ceiling for the number of possible LLM provider calls the system makes, which scales with your number of inputs.

Code examples

There are two ways to use Expected output mode:

With the optimize_from_ground_truth_options method.
Or From an optimization config.

From options

To use the Expected output mode, use the optimize_from_ground_truth_options method and include a ground_truth_responses array that collates the inputs together rather than the raw lists we use otherwise.

Here’s how:

Python

1 # ...
2 options = GroundTruthOptimizationOptions(
3         context_choices=[ld_context],
4         max_attempts=5,
5         model_choices=["claude-opus-4-5", "claude-haiku-4-5"],
6         judge_model="claude-opus-4-5",
7         handle_agent_call=handle_llm_call,
8         handle_judge_call=handle_llm_call,
9         judges={
10             "relevance": OptimizationJudge(
11                 judge_key="my-relevance-judge",
12                 threshold=0.8,
13             ),
14         },
15         ground_truth_responses=[
16             GroundTruthSample(
17                 user_input="I'm going to tokyo next week, where should I stay near Shinjuku?",
18                 expected_response="Recommend hotels in the Shinjuku area of Tokyo.",
19                 variables={"city": "Tokyo", "region": "Shinjuku"},
20             ),
21             GroundTruthSample(
22                 user_input="where to eat in anchorage",
23                 expected_response="Recommend restaurants in Anchorage, Alaska.",
24                 variables={"city": "Anchorage", "region": "Downtown"},
25             ),
26             GroundTruthSample(
27                 user_input="airbnbs near tahoe",
28                 expected_response="Recommend Airbnb rentals near Lake Tahoe.",
29                 variables={"city": "Lake Tahoe", "region": "South Shore"},
30             ),
31             GroundTruthSample(
32                 user_input="what are some food options in sf near the airport",
33                 expected_response="Recommend restaurants near SFO in San Francisco.",
34                 variables={"city": "San Francisco", "region": "SFO"},
35             ),
36         ],
37     )
38 
39 # ...
40 
41 result = await optimization_client.optimize_from_ground_truth_options("your-agent-key", options)

From an optimization config

When operating from a config directly, LaunchDarkly automatically infers the type of invocation you wish to use. The implementation configuration is the same regardless of which invocation occurs.

Here’s how:

Python

1 result = await optimization_client.optimize_from_config(
2     "my-optimization-config-key",
3     OptimizationFromConfigOptions(
4         project_key="my-project",
5         context_choices=[ld_context],
6         handle_agent_call=handle_llm_call,
7         handle_judge_call=handle_llm_call,
8     ),
9 )