> ## Documentation Index
> Fetch the complete documentation index at: https://launchdarkly.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Cheaper per success, worse for users: testing tool search on an LLM agent

> Tool search cut an agent's tokens per task by 78% and its success rate from 71% to 27%.

**Developer**

**Federal docs**

**EU docs**

*Published January 19, 2027*

Portrait of Scarlett Attensil.

by Scarlett Attensil

In [Part 1](https://launchdarkly.com/docs/tutorials/llm-p90.md), Claude Haiku 5.5 came out as the cheaper model per task completed within 30 seconds. It also failed one task in four, and every request carried all 128 tool definitions. Anthropic's [tool search](https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool) looks like the fix for both. It sends the tool definitions with `defer_loading: true`, keeps them out of the model's context, and lets the model search for the ones it needs. The docs say tool search typically cuts definition tokens by over 85% and that tool selection degrades once a model has more than 30 to 50 tools to choose from.

So I kept the model fixed and changed one thing. I ran 170 tasks through a LaunchDarkly Agent Management experiment, each assigned at random to one of two Haiku variations: all 128 tools up front, or the same tools behind tool search.

Tool search did what it promised on cost. Tokens per task fell 78%, and each task completed within 30 seconds cost less than with all tools loaded. It also cut the share of tasks completed successfully from 71% to 27%.

Both of those numbers are true. Only one of them describes what users get.

**What we found**

* **Tool search cut tokens per task by 78%.** The experiment measured 538,515 tokens per task with all tools up front and 116,849 with tool search.
* **It cut successful completion from 71% to 27%.** Haiku completed 58 of 82 tasks with all tools and 24 of 88 with tool search. Within 30 seconds, the counts were 57 and 19. Both drops are statistically significant.
* **Cost per success favored the variation that failed more.** Each task completed within 30 seconds cost about \$0.06 with tool search and \$0.08 to \$0.39 with all tools. Judged on cost per success, tool search wins. Judged on successes per task assigned, it loses badly.
* **Many failures traced back to missed searches.** In the traced failures, a search did not return the tool the task needed, and the model asked the user or said it could not help instead of searching again. With tool search, 19 failed tasks failed on a turn with no tool call, against 3 with all tools loaded.
* **The result replicated.** An earlier run of the same experiment, on separate request contexts, gave 77% against 29% and a 77% token cut.

This tutorial runs that experiment in [LaunchDarkly Agent Management](https://launchdarkly.com/docs/home/agentcontrol) on the project from Part 1, shows how to defer tools from the config, and reads cost and success on one results page.

## What you will build

You add a second Agent Management config, `bfcl-tool-search`, to the `llm-p90-experiment` project. It holds two Claude Haiku 5.5 variations with the same system message, parameters, and 128 tools. The only difference is one model parameter, `tool_search: bm25`, on the second variation. A 50/50 experiment on the `request` unit compares them on four metrics: success rate, tasks completed within 30 seconds, tokens per task, and p90 among successes.

Everything else comes from Part 1: the Berkeley Function Calling Leaderboard (BFCL) tasks, the checker, the runners, the `request` context kind, and the success, 30-second, and duration metrics.

## Prerequisites

Complete [Part 1](https://launchdarkly.com/docs/tutorials/llm-p90.md) through Step 3. The project, tools, `request` context kind, and the `task-resolution`, `task-resolved-within-30s`, and `task-resolution-duration-p90` metrics must exist. Pull the latest [companion repository](https://github.com/launchdarkly-labs/scarlett-llm-p90) and run `uv sync`.

## Step 1: Switch tool search on from the config

The two variations should differ in exactly one setting, and that setting should live in the config, not in code. Tool search needs two changes to the request: a tool search tool first in the `tools` array, and `defer_loading: true` on every other tool. At least one tool has to stay loaded, and the search tool counts.

The repository's `llm_p90/tool_search_handler.py` makes those changes when the variation's model parameters include `tool_search`:

```python lines wrap theme={null}
def build_tools(config):
    attached = config.get("tools") or {}
    mode = tool_search_mode(config)  # "bm25", "regex", or None
    tools = [
        {
            "name": name,
            "description": tool.get("description", ""),
            "input_schema": tool.get("parameters", {}),
            **({"defer_loading": True} if mode else {}),
        }
        for name, tool in attached.items()
    ]
    return [dict(SEARCH_TOOLS[mode]), *tools] if mode else tools
```

Both arms run through this handler. The only difference between them is the parameter. The handler reuses the Agent Management Claude handler's own prompt building and parameter forwarding, and `tool_search` is not on the forwarding list. The key never reaches the model provider. The handler also carries each task's full message content from one turn to the next. In a later turn, the model can reuse a tool it found earlier without searching again, which matches how an application that passes back `response.content` behaves.

Create the config. Paste this into a coding agent with the [LaunchDarkly hosted Model Context Protocol (MCP) server](https://launchdarkly.com/docs/home/getting-started/mcp-hosted.md) and skills:

```text lines wrap theme={null}
Use the LaunchDarkly configs-create skill. In project llm-p90-experiment,
create Agent Management config bfcl-tool-search in completion mode with two
variations, all-tools and tool-search. Copy the model, max tokens, system
message, and all attached tools from the claude-haiku-5-5 variation of
bfcl-model-comparison. On tool-search only, add the model parameter
tool_search with the value bm25.
```

The same setup runs from one command with `uv run python scripts/create_tool_search_config.py`, which also reads both variations back and checks every field.

## Step 2: Measure cost and success on the same page

Cost per success is a ratio, and a ratio can improve because its denominator shrank. To keep that visible, put success and cost on the experiment as separate metrics. The AI SDK already reports token usage to Agent Management after every successful generation. Tokens need no custom event. Ask the agent:

```text lines wrap theme={null}
Use the launchdarkly-metric-create and launchdarkly-experiment-setup skills.
In project llm-p90-experiment, create:

- Metric tokens-per-task: custom value on event $ld:ai:tokens:total,
  analysis unit request, sum per unit, exclude units without events,
  lower is better, unit tokens
- Experiment bfcl-tool-search in the test environment on the Agent Management
  config bfcl-tool-search: all-tools baseline, tool-search treatment,
  50/50, randomization unit request, task-resolution primary,
  task-resolved-within-30s, tokens-per-task, and
  task-resolution-duration-p90 secondary

Show me the design before you create the experiment. Leave it as a draft.
```

Success is the primary metric because it is the one a token saving can trade away. The run shown in Step 4 had p90 among successes as its primary metric. That metric is the first row of its results table. The runner gives this experiment its own request contexts, `bfcl-tool-search-<task>`. Its events stay separate from the Part 1 experiment.

## Step 3: Check one task, then run 170

Check one task on both arms first. Compare the input tokens on the two evaluation runs. The tool-search arm should use far fewer:

```bash lines wrap theme={null}
uv run python run_evaluation.py --config bfcl-tool-search --tasks 1
```

In this run the tool-search arm used 20,770 input tokens on that task and the all-tools arm used 222,431. If the two numbers are close, tool search is not switched on. Check that `tool_search` is a model parameter on the variation, not a custom parameter.

Send five tasks through the experiment runner and confirm that all four metrics show recent events, as in Part 1. Then start the iteration and run the 170:

```bash lines wrap theme={null}
uv run python run_experiment.py \
  --config bfcl-tool-search \
  --tasks 170 \
  --output logs/tool_search_confirmatory.jsonl \
  --confirm-paid-run
```

Each row of the log records the arm, the number of tool searches the model ran, and whether the task completed successfully. To repeat the experiment in a new iteration, add `--run-id` with a new label and a new `--output` file. Every task then gets a fresh request context, and LaunchDarkly assigns it again.

## Step 4: Read cost and success together

![The bfcl-tool-search results table, iteration 2 running. task-resolution-duration-p90, the primary metric: Haiku 5.5 all tools up front 21,494.93 ms and Haiku 5.5 tool search 39,111.68 ms, relative difference plus 81.96% with an interval of minus 58.10 to plus 273.23, p-value 1.0000. Task resolution: all tools 70.73% and tool search 27.27%, relative difference minus 61.44% with an interval of minus 75.65 to minus 47.23, p-value 0.0000.](https://mintcdn.com/launchdarkly/YcqpuW95bNnoR9KX/images/tutorials/llm-tool-search/tool-search-results.png?fit=max\&auto=format\&n=YcqpuW95bNnoR9KX\&q=85\&s=8875fec6b1dd48d0df453ff1bac6c2ea)

Results for the rerun iteration. Captured in the project's production environment; the steps in this tutorial use `test`.

**Success rate.** All tools up front completed 58 of 82 tasks successfully (70.7%). Tool search completed 24 of 88 (27.3%). The experiment puts the relative difference at minus 61.4%, with an interval of minus 75.7 to minus 47.2%.

**Completed within 30 seconds.** 57 of 82 against 19 of 88: 69.5% and 21.6%, a relative difference of minus 68.9%, with an interval of minus 82.1 to minus 55.8%.

**Tokens per task.** 538,515 against 116,849, a relative difference of minus 78.3%, with an interval of minus 82.1 to minus 74.5%. All three differences are statistically significant.

**P90 among successes.** All tools up front came in at 21.5 seconds and tool search at 39.1. The relative difference is plus 82%, with an interval of minus 58 to plus 273%. The difference is not statistically significant. Look at the sample sizes: the row for p90 among successes covers 58 and 24 requests. That makes it a comparison between different sets of tasks. Tool search completed the 24 it could, which are likely the easier ones. If anything, the row flatters tool search. The two runs back that up. Each run assigned the same 170 tasks at random. Some tasks landed on tool search in one run and on all tools in the other. The tasks tool search completed succeeded 88% of the time with all tools (23 of 26). The tasks tool search failed succeeded 67% of the time with all tools (40 of 60). The interval runs from minus 58 to plus 273%, and in the earlier run the same row read minus 1%. Read it as a description of the successes, as in Part 1, and decide on the two success metrics.

**Cost.** The AI SDK tracks tokens once per successful generation, which here is one user turn of one attempt. The handler can make several provider requests inside a generation. With all tools up front, 420 generations used 44.0 million input tokens, about 105,000 per generation. With tool search, 601 generations used 10.0 million, about 17,000 per generation. Claude Haiku 5.5 [prices each request by its own prompt length](https://platform.claude.com/docs/en/about-claude/pricing#long-context-pricing): \$0.10 per million input tokens up to 100,000, and \$0.50 above. An all-tools generation's requests can fall on either side of that line. The range below prices them all at the standard rate for the low end and all at the higher rate for the high end. Tool-search generations average about 17,000 input tokens, far below it. The table prices them at the standard rate.

<table>
  <thead>
    <tr>
      <th />

      <th>All tools up front</th>
      <th>Tool search</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Tokens per task</td>
      <td>538,515</td>
      <td>116,849</td>
    </tr>

    <tr>
      <td>Cost per task</td>
      <td>\$0.055 to \$0.27</td>
      <td>\$0.013</td>
    </tr>

    <tr>
      <td>Tasks completed within 30 seconds</td>
      <td>57 of 82</td>
      <td>19 of 88</td>
    </tr>

    <tr>
      <td>Cost per task completed within 30 seconds</td>
      <td>\$0.079 to \$0.39</td>
      <td>\$0.061</td>
    </tr>
  </tbody>
</table>

Read the last row alone and tool search is the cheaper way to get an on-time success. Read the row above it and tool search delivers an on-time success to about one user in five, against more than two in three with all tools loaded. A cost-per-success report rewards the variation that fails more, because failed tasks are cheap. The decision belongs to successes per task assigned, next to cost per task assigned.

## Step 5: Find out why the searches failed

The checker's error messages show where tasks broke. With tool search, 19 failed tasks failed on a turn where the model made no tool call at all, against 3 with all tools loaded. Failed tool-search tasks ran a median of six searches, counted across both attempts and every turn. The model searched and then stopped. The retry did not rescue many: tool search completed 12 of 88 tasks on the first attempt and recovered 12 of the 76 that failed once.

To see why, the handler writes a trace of every response when `LLM_P90_TRACE` points to a file:

```bash lines wrap theme={null}
LLM_P90_TRACE=trace.jsonl uv run python run_evaluation.py --config bfcl-tool-search --tasks 1
```

The trace records each search query, the tools it returned (five per search by default), and what the model did next. I replayed five failed tasks from the first run with the trace on. None showed a search error or a response cut off. Four of the five failed the same way, and the fifth made its calls but left the vehicle system in the wrong state:

* A user asked to fly from Rivermist to Los Angeles. The model searched for "search flights origin destination date seating class" and found `get_flight_cost` and `book_flight`. The task also needed `list_all_airports` to turn "Rivermist" into an airport code, and no search returned it. The model asked the user for the airport code and ended the turn.
* A later turn asked for a social media post. The model searched for "social media post update share account login" and got five trading tools. It replied that it had no tool for posting. The Twitter tools were in the catalog, deferred, and not returned by that search.

Once one turn fails, the next turns fail with it. The user asks to cancel the flight, and the model correctly says nothing was booked.

With all 128 tools loaded, both of those tools sit in the model's context. With tool search, a task depends on each search returning the right tool, and on the model searching again when it does not.

## What this result covers

This experiment tests tool search in its strictest form: every tool deferred and a neutral one-line system message. Anthropic's [optimization tips](https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool#optimization-tips) recommend two things this run did not do: keep your three to five most-used tools loaded, and add a system prompt section that describes the available tool categories. The BFCL tool descriptions also share long boilerplate openings, which gives a keyword search less to separate them by. A setup that follows those tips is the next variation to test, and the same experiment measures it. Add a third variation and compare all three on success within 30 seconds and cost per task.

Treat the result as a property of this setup, not of tool search in general. The method is the general part.

## Ship the cheaper variation only if success holds

Suppose the cost row had won you over and you shipped tool search. In Agent Management that is a [guarded rollout](https://launchdarkly.com/docs/home/releases/guarded-rollouts) on the config: serve the tool-search variation to a growing share of requests, with `task-resolution` as the guardrail. A success rate falling from 71% to 27% is the kind of regression the guardrail is there to catch, and the rollout can [roll back automatically](https://launchdarkly.com/docs/home/releases/managing-guarded-rollouts#automatic-rollback-behavior) before most users see it. The token savings would show up on the **Monitoring** tab either way. The guardrail is what keeps them from being bought with failed tasks. Guarded rollouts are available on select plans, and a rollout and an experiment do not run on the same config at the same time.

## What this means for cost decisions

Part 1 showed that a latency percentile computed on successes describes only the successes, while the failure rate moves beside it. Part 2 shows the same thing for cost. Cost per success divides by the successes. A change that fails more often can make every remaining success cheaper.

Before you ship a change that saves tokens:

1. **Put success rate and cost per task on the same experiment.** Agent Management tracks the tokens automatically. The success event is the one you add.
2. **Divide by tasks assigned, not tasks completed.** Cost per task assigned and successes per task assigned can be compared directly. Cost per success cannot be read alone.
3. **Trace a few failures before you trust a big swing.** A large drop can be a harness bug or real behavior, and the trace tells you which.
4. **Guard the rollout on success.** Use the success metric as the guardrail when you ship. A saving that costs completions then rolls back on its own.
