Skip to main content
I wanted to know whether I could switch a tool-calling agent to a cheaper model without making users wait longer. I set it up as an experiment in LaunchDarkly Agent Management: Claude Sonnet 5.5 and Claude Haiku 5.5 as two variations of one config, 170 multi-step tasks, and each task assigned at random, 85 to each model. For latency, the usual advice is to compare a tail percentile rather than the average because an average hides the slow tail. The LaunchDarkly metric reports p90 among successes: the 90th percentile of durations from tasks that send a duration event. I looked at both. Each task is a short conversation with an agent that has 128 tools. In one, the user asks for the fare between the first two airports in the system’s list, then asks to book a flight from San Francisco to Los Angeles with a given card, then asks for travel insurance on that booking. After the conversation, a checker replays the calls and compares the simulated travel system with the expected state after every turn: the booking, the insurance on it, the card balance. A task counts as completed successfully when the checker accepts every turn, within two attempts. Anything else counts as failed. Duration covers the whole task, from the first model call through validation, retry included. The average picked Haiku: 17.3 seconds per successful task, against 18.5 for Sonnet. The p90 among successes tipped the other way. Sonnet came in at 28.9 seconds and Haiku at 29.5, a gap of about half a second and well inside the noise. The tail-aware number took away the speed edge the average had given Haiku. The input-token prices did not move. For prompts up to 100,000 tokens, Haiku 5.5 lists at $0.10 per million input tokens to Sonnet 5.5’s $2, a twentieth of the input-token price. With estimates of p90 among successes this close, that much room to save made switching look like free money. Then I counted the failures. Haiku failed 22 of its 85 tasks, about one in four. Sonnet failed 14, about one in six. Three questions follow. How was p90 among successes measured? This experiment recorded a task’s duration only when the task completed successfully, and percentile analysis in LaunchDarkly excludes requests that send no event for the metric. The 28.9 and 29.5 seconds are the p90 among 71 Sonnet successes and 63 Haiku successes. The failures are counted by the success-rate metric instead. How do you know about the failures? The same results page shows them twice. The success-rate metric counts every assigned request and scores a failure as zero: 83.5% for Sonnet and 74.1% for Haiku. The row for p90 among successes has sample sizes of 71 and 63, not the 85 requests each model was assigned. If a percentile’s sample size is smaller than the number of requests assigned, the percentile describes only part of the traffic. Can you rely on p90 among successes? For what it measures, yes. It shows how fast each model was when it succeeded, and here the two were too close to tell apart. It does not show p90 of time to successful completion: the time by which 90% of assigned tasks have succeeded, with failures counted as never finishing. Neither model ever reached 90% success, and under the two-attempt limit a failure is final. That p90 of time to successful completion does not exist. Decide on success rate and on success within a time budget, and read p90 among successes as a description of the successes. This tutorial builds that experiment in LaunchDarkly Agent Management: one config holding both models, an outcome recorded for every assigned task, and success rate, p90 among successes, and cost compared on one results page. What we found
  • The average and p90 among successes pointed at different models. The average made Haiku look faster. The p90 among successes erased that edge and tipped, within noise, to Sonnet. Neither number counted the failures.
  • Capable models fail a lot of multi-step work. Within two attempts, Sonnet 5.5 failed 16% of its tasks and Haiku 5.5 failed 26%.
  • A latency percentile describes only the attempts that report a duration. Dropping failures skews a comparison at any failure rate. Once more than 10% of tasks fail, a p90 of time to successful completion stops existing, and neither model has one here. At exactly 10%, the answer depends on the quantile convention.
  • The comparison that holds up is success within a time budget, next to cost. Within 30 seconds, Sonnet completed 76% of its tasks successfully and Haiku 68%. Token tracking in Agent Management shows that each of those on-time successes cost about $1.33 on Sonnet and between $0.08 and $0.39 on Haiku.
  • The experiment puts uncertainty around an apparent gap. Sonnet’s 9-point edge in success rate looks decisive. LaunchDarkly reports it as not significant at 170 tasks, with a p-value of 0.11. Shipping on the raw counts would have been a guess.

What you will build

The companion repository has the tasks, the checker, and the runners. You add API keys. You end up with these pieces, all in a project named llm-p90-experiment:
  • One config holding both models. Claude Haiku 5.5 and Claude Sonnet 5.5 are variations with the same 128 tools and parameters, and the application reads whichever one it is served.
  • An experiment that assigns each request at random. Both arms draw from the same task pool, and interleaving reduces the chance that time drift favors one model.
  • Success, latency, and cost on one results page. A conversion metric for success, a metric for p90 among successes, a 30-second metric, and the token usage the AI SDK reports automatically.
  • A Python runner that plays each task, scores it with the official checker, and sends the events.

Watch the full setup and results walkthrough.

Why a multi-step workload, and why BFCL

Counting failures requires a definition of “done.” Most production telemetry cannot: the provider returned a 200, the model returned text, and nobody checked whether the job got done. The Berkeley Function Calling Leaderboard (BFCL) can. Each multi-turn task has the model call functions against simulated backends, a file system, a trading account, a ticketing system, across several user turns, and an official checker compares the end state with the ground truth. Every failure is known exactly, and anyone can rerun the same 200 public tasks. Multi-step work is also where failures pile up. In this run, tasks whose correct solution takes longer tool-call sequences were less likely to succeed: across both models, 37 of 41 tasks needing one to four calls completed successfully (90%), 85 of 111 needing five to eight (77%), and 12 of 18 needing nine or more (67%). The leaderboard shows a similar gap. In the pinned BFCL V4 snapshot, the previous generation of Claude models scores 86% to 89% on BFCL’s single-turn categories, scored by abstract syntax tree (AST) matching and labeled Non-Live AST, and 54% to 68% on multi-turn. The multi-turn checker also inspects system state. Treat that gap as context, not a controlled test of adding calls to the same task. BFCL is a stand-in for the agent work you actually ship: tickets, checkouts, code edits. To run this on your own traffic, swap the JSONL files for your requests and replace the BFCL check in llm_p90/scoring.py (evaluate_with_bfcl) with whatever “done” means in your product. Keep the three events: task-resolved when a task completes successfully, task-resolution-duration-ms for the time from the first model call to that moment, and task-resolved-within-slo when it completed inside your time budget. The LaunchDarkly experiment does not know or care that Berkeley wrote the checker.

Prerequisites

You need Python 3.12 or later and uv. You also need LaunchDarkly Agent Management and Experimentation, the LaunchDarkly hosted Model Context Protocol (MCP) server at https://mcp.launchdarkly.com/mcp connected to your coding agent, and the LaunchDarkly agent skills from launchdarkly/ai-tooling. Create an API access token with project write access for tools, configs, and evaluations, plus an ANTHROPIC_API_KEY. Step 3 creates the project if needed, then you copy its test server-side SDK key. For Cursor Agent CLI, add the hosted server to .cursor/mcp.json, authenticate, and verify the connection:
Install the agent skills into your coding agent with one command:
A full pass is 30 paired pilot tasks, a 5-task smoke check, then 170 randomized experiment tasks. Step 4 shows the bill before you commit to the 170.

Step 1: Why p90 of time to successful completion can disappear

A p90 of time to successful completion is a seat in a sorted list. Sort 100 task outcomes and read the value 90% of the way along, counting failures as never finishing. If 19 tasks fail, seat 90 is empty. The chart still prints p90 among successes. It sorted the 81 observed durations and reported seat 73 of that shorter list. A retry does not fill empty seats by magic. If each attempt succeeds independently with probability p, two attempts leave (1-p)² failed. Independence is the flattering case. The tasks that fail once are the hard ones, which means the second attempt is worse than a fresh draw. The pinned BFCL V4 snapshot lists multi-turn base accuracy with native function calling, the category this run uses, of 63.50% for Claude Haiku 4.5 and 69.00% for Claude Sonnet 4.5. Square the failure rates and you get roughly 13% and 10% still failed after two tries. Count those as final and the cheaper model has no p90 time to successful completion. The stronger one sits on the 10% line, where a finite sample cannot tell you which side it is on. Drop the failed tasks from the sample and you have changed the question. Those are the previous generation of the same cheap-and-fast versus strong pairing, on Berkeley’s harness, region, and week. This tutorial runs their successors, Claude Haiku 5.5 and Claude Sonnet 5.5, which have no BFCL scores yet, and BFCL gives each task only its own tools while this run attaches all 128. So the numbers are a prior, not a prediction of your run. They are enough to refuse a single primary metric of “p90 of time to successful completion.” Use three metrics instead:
  • Resolution: Share of assigned requests that complete successfully within two attempts. A missing event counts as zero.
  • P90 among successes: 90th percentile whole-task time among the requests that completed successfully.
  • Resolved within 30 seconds: Share of assigned requests that complete successfully inside a time budget. Missing and slow both count as zero. This metric exists however many tasks fail.
Read resolution first. A faster p90 among successes does not offset a lower success rate. The 30-second metric, next to cost, is the comparison Step 6 ends on.
One row of 100 tasks sorted fastest to slowest. 81 cells are filled and 19 are gray. A p90 marker points at empty seat 90, and a second marker points at seat 73, labeled what the chart reports. Illustrative 19%, not a measured result.

Seat 90 has no observation. P90 among successes is seat 73 of the successes.

llm_p90.analysis puts a Wilson interval on the unresolved rate. Interval wholly below 10%: a finite p90 of time to successful completion is supported. Wholly above: it is not. Crossing 10%: the sample cannot say. At exactly 10% unresolved, the answer depends on the quantile convention. This repository uses the type-7 quantile from Hyndman and Fan (1996), under which that boundary is infinite.

Step 2: Clone the repository and check it offline

The tasks pin to Gorilla commit 6ea57973c7a6097fd7c5915698c54c17c5b1b6c8. The export hashes source ids, sorts, and writes 30 pilot rows plus 170 experiment rows with no overlap. data/provenance.json stores the hashes.
The tests run with no keys. They cover the BFCL adapter, the analysis boundary, the dataset hashes, and the interactive harness: a scripted model that plays each task’s ground truth resolves all 200 tasks through the same loop the paid runs use, and both runners are driven end to end with a fake provider. uv run ruff check . should report no findings.
Fill LD_OPERATOR_TOKEN and ANTHROPIC_API_KEY. LD_OPERATOR_TOKEN is the LaunchDarkly API access token used to create tools, attach them to variations, and write evaluation runs. Leave LD_PROJECT_KEY=llm-p90-experiment. You add LD_SDK_KEY after creating the project in Step 3.

Step 3: Create the config, metrics, and experiment

This step uses two small scripts and four agent prompts. The companion repository contains the scripts and pinned inputs. First, make sure the project exists:
Copy the test environment’s server-side SDK key into LD_SDK_KEY in .env.

Create the tools and the config

tools/bfcl_tools.json holds BFCL’s 128 multi-turn functions for eight simulated backends, converted to JSON Schema by BFCL’s own convert_to_tool. The file trims two descriptions to fit the LaunchDarkly 1,024-character description limit and uses lowercase snake_case keys, the required format for tool keys. This creates them in Agent management → Library → Tools and skips any that already exist:
Then create the config. Paste this into a coding agent that has the LaunchDarkly MCP server and skills:
Attach all 128 tools to both variations:
Everything the model sees comes from the config: the model, its parameters, and the tool definitions. The provider handler sends the attached tools through Anthropic’s native tools parameter, and the runners supply only the code that executes each call on BFCL’s backends. To change a model or a tool, edit the variation in Agent Management. The next request picks it up. Every task gets all 128 tools. Picking the right one out of the clutter is part of the test. BFCL itself offers each task only its own backends’ tools, about 28. Anthropic’s tool search could defer most definitions and load them on demand, but that turns a failure into either a missed search or a bad call. That is a separate experiment, and it is Part 2 of this series. The variations set no temperature, because Claude models from 4.7 on reject a non-default one. A completion variation carries at least one message. The system message is a single neutral line that adds no task guidance. Under Agent management, select Agents, open the Completions tab, select bfcl-model-comparison, and check both variations: model, max tokens, and the attached tools.
Agent Management with two active bfcl-model-comparison variations. Sonnet is expanded, showing max_tokens 2048, the one-line system message, and the attached versioned tools.

The current Agent Management config page. Captured in the project's production environment; the steps in this tutorial use test.

Register the request context kind

A new project knows one context kind, user. The metrics and experiment randomize by request, and LaunchDarkly creates a context kind the first time an SDK evaluates a context of that kind. This script evaluates bfcl-model-comparison once for a request context. The script never calls a model provider and costs nothing. Here it is:
Then confirm where it landed. Ask the agent:
The context should be in test. If it shows up in production, your LD_SDK_KEY is the production key. Swap in the test key and run the script again. Every event in Steps 4 and 5 follows that key.

Create the metrics and the experiment

Back in the agent. The skills should print the design and wait. Do not let them start an iteration.
The read-back matters. An average duration metric would look fine on the chart and answer a different question. The LaunchDarkly guide to proving ROI with AI experiments measures latency at a percentile for the same reason. If the agent reports that request is not an available randomization unit, mark it available as the README describes. Open the metrics. Resolution includes every assigned request. P90 among successes includes the ones that sent task-resolution-duration-ms. Percentile analysis works that way on purpose: a percentile is defined on observed values. The conversion metric is how unfinished work stays in the experiment.
Two current metric detail views. Task resolution is the percent of request contexts that sent task-resolved. Task resolution duration averages task-resolution-duration-ms events per request, then computes p90 among successes in milliseconds.

The current resolution and p90 among successes metric definitions. Captured in the project's production environment; the steps in this tutorial use test.

The Python AI SDK already records generation duration, tokens, success, and error for every variation, with no extra code. Those metrics fill the config’s Monitoring tab and supply the cost figures in Step 6. They describe model calls, not tasks: a 200 OK from the provider is not a BFCL pass, and one task can span several calls. That is why the task-level success and duration events sit alongside them.

Step 4: Run a 30-task pilot

The pilot sends each of 30 tasks to both models once through evaluations from code. Each row is a whole multi-turn task. A wrapper handler plays it turn by turn, executing every call against BFCL’s backends, and the official checker scores the result. A task takes several model calls, not one. Pairing is the right design for an offline calibration: task difficulty dominates the variance. Those 30 rows never enter the experiment analysis. Check one task on both models first. It is the cheapest way to find a wrong key or a missing tool before you pay for 30:
You get one evaluation result object per model. Score 1 means the official checker accepted the end state of every turn. Under Agent management, select Evaluations, then Evals from code and open a run. You want at least one 0. All 1s usually means the scorer never ran. All 0s is often bad JSON before it is a bad model.
Evaluation run summary showing 30 rows processed, 22 passed, and 8 failed. The results table compares each input, expected output, model output, and official BFCL score.

A 30-row calibration run in the current Evaluations UI. Your five-row pilot uses the same summary and results table.

Evaluations keep their scores on the run itself. Experiment metrics come from the events your application sends. The pilot leaves the experiment’s metrics empty by design. Step 5 confirms those events arrive before the paid run. After this step, the usage page for your model provider shows what the 62 pilot task runs cost. A task is many model calls, and each call carries all 128 tool definitions. Price the 170 from this number, not from a guess.

Step 5: Prove the events land, then run 170

Open the draft experiment first.
The current bfcl-model-comparison design: all request contexts targeted, Haiku and Sonnet split 50/50, task resolution primary, task-resolution-duration-p90 attached, and frequentist sequential testing at a 0.05 significance level.

The same design after the measured run started. Before you start yours, the status bar should say Ready to start.

Send five pilot tasks through the experiment runner, to a log that is not the results file:
Until the iteration starts, every task gets the default variation, which is fine for a smoke check. The process prints resolved=True or resolved=False per task. The three custom task events fire only on a success, and task-resolved-within-slo requires that success to take no more than 30 seconds. You need at least one success inside 30 seconds to observe all three keys. If you do not get one, rerun with --tasks 10 --output logs/smoke2.jsonl. The first five repeat and the next five are new. Exposure and autogenerated AI telemetry still arrive for every SDK evaluation.
The MCP server reports a last-seen timestamp for each event key. Alternatively, open Metrics and read the Events seen column for all three metrics. A missing key is usually LD_SDK_KEY pointed at the wrong environment. The SDK will accept the events anyway. They just land somewhere you are not looking. Start the iteration. Before spending the full budget, ask the agent:
Then:
--confirm-paid-run is required because this spends provider budget. For each task, LaunchDarkly picks the variation. The runner plays the task turn by turn with that model, executing each call against BFCL’s backends, then scores the run with the official checker. On failure it restarts the task once from its initial state, with the checker’s error type and message added to the first turn. For method-order errors, BFCL’s message can name a missing tool method. It never includes the expected arguments, output values, or judge critique.
Failed tasks send none of these events. Resolution and the 30-second metric treat them as zero. The metric for p90 among successes, by design, measures only the requests that report a duration. The runner writes every task, including failures, to logs/confirmatory.jsonl. Unresolved counts in Step 6 come from that file. If the process stops after writing N rows, keep the same iteration and request keys. Resume the remaining rows in a new output file:
Replace N with the line count of the first output. Keeping the same request keys preserves each task’s original assignment. Combine the two logs before running the local analysis.
Concurrency is 1. No temperature is set. Max tokens is 2048. protocol.json also records streaming off and prompt caching off. Provider calls use the Anthropic Python SDK defaults: a 10-minute timeout and up to 2 automatic retries on rate limits, server errors, and dropped connections. Retry time counts toward the task’s duration, the same way it would for a user waiting on the answer. An error that survives the retries stops the run, and no events are sent for that task.All 128 tool definitions go out on every call, with no tool search and no prompt caching. That’s why the bill grows with tool count, and it’s the first thing to change if you run this on your own traffic.The 170 is fixed before the first confirmatory call. We do not stop early. LaunchDarkly still analyzes percentile metrics with frequentist sequential methods, which is why opening the results page mid-run does not invalidate the estimator. It also does not change the stopping rule. The in-app sample size calculator is built for fixed-horizon means. Size a percentile run like this one from the pilot cost.Request randomization is here because this is a production-shaped experiment. Interleaving makes it less likely that time drift favors one arm, but it does not guarantee identical task mixes. For a purely offline bake-off, the paired pilot design is more efficient. Do not describe the 170 as paired. The experiment analysis in LaunchDarkly treats each request as independent, not paired.

Step 6: Read the results

Open the experiment after the runner exits. Resolution first, then p90 among successes. The companion repository includes the 170 source rows, the data/confirmatory_results.json aggregates, and a LaunchDarkly resource snapshot.
The bfcl-model-comparison results table, iteration running, 170 request contexts split 50/50. Task resolution, the primary metric: Claude Sonnet 5.5 83.53% and Claude Haiku 5.5 74.12%, relative difference minus 11.27% with an interval of minus 25.21 to plus 2.68, p-value 0.1132, sample size 85 each. task-resolution-duration-p90: Sonnet 28,939.53 and Haiku 29,459.23 milliseconds, relative difference plus 1.80% with an interval of minus 69.22 to plus 139.51, p-value 1.0000, sample sizes 71 and 63.

Pinned results for the 170-task iteration, captured before a later unrelated exposure. This screenshot matches the hashed results artifact.

Resolution. Sonnet completed 71 of 85 tasks successfully (83.5%) and Haiku 63 of 85 (74.1%). That is 11.3% lower for Haiku, with an interval of minus 25.2% to plus 2.7% and a p-value of 0.11. The interval leans toward Sonnet but still includes zero. At 170 tasks, the run does not settle which model succeeds more often. That inconclusive result is useful: it stopped us from declaring a winner that the data did not support. Without the interval, a 9-point gap reads as a winner. P90 among successes. Sonnet came in at 28.9 seconds and Haiku at 29.5, with an interval from minus 69% to plus 140%. Look at the sample-size column: the rows for p90 among successes count 71 and 63, not 85 and 85. The 22 tasks Haiku failed, and Sonnet’s 14, are absent. (summarize_trials gives Haiku 29.3 seconds from the same log. It uses the type-7 quantile, and the percentile estimator in LaunchDarkly lands slightly differently.) The same comparison is available from your coding agent through the MCP server. Ask:
The last line is the one that matters here. The agent reports that the metric for p90 among successes analyzed 71 and 63 requests out of 85 assigned to each model. The percentile values themselves are on the results page. What a latency chart would have told you. Close estimates of p90 among successes, or a nudge toward Haiku: its average time among successful tasks was 17.3 seconds against Sonnet’s 18.5. Count failures as never finishing and neither model has a p90 of time to successful completion at all. The failure rate is 25.9% for Haiku (Wilson interval 17.8 to 36.1) and 16.5% for Sonnet (10.1 to 25.8), and both intervals sit wholly above 10%. What to compare instead. Pick a time budget that stands for a good experience and count the tasks that completed successfully inside it. Failed and slow tasks both count as misses. This number always exists. With the Step 3 prompt, task-resolved-within-30s sits on the results page next to the other two. In the run shown here, the metric was added after the fact. These figures come from logs/confirmatory.jsonl:
Resolved withinHaiku 5.5Sonnet 5.5
20 seconds48%54%
30 seconds68%76%
45 seconds73%81%
Now put cost next to it. The AI SDK reports token usage to Agent Management after every model call. The config’s Monitoring tab already shows tokens and cost per variation. The smoke check in Step 5 lands in those totals too. To isolate the 170-task run, query config metrics by variation with the run’s start and end times. For this run:
Haiku 5.5Sonnet 5.5
Generations, one per user turn427390
Tokens per taskabout 518,000about 503,000
Cost per task$0.05 to $0.27$1.02
Cost per task resolved within 30 seconds$0.08 to $0.39$1.33
Both models read about the same number of tokens per task. Most of that is the 128 tool definitions, resent on every call. So the cost gap is price, not volume. The Haiku range exists because Claude Haiku 5.5 is priced by prompt length: a request whose prompt is over 100,000 tokens pays $0.50 per million input tokens instead of $0.10. Each generation here is one user turn, and the provider handler can make several requests inside it. The roughly 100,000 tokens per generation are spread across several prompts. Anthropic prices each request by its own prompt length. The low end assumes every request is under the threshold, and the high end assumes every request is over it. Your provider’s per-request usage shows the exact split. That turns the result into the decision you actually face. In this run, Sonnet completed about 8 more tasks per 100 successfully inside the time budget, and each on-time task cost three to seventeen times as much. The 30-second comparison is descriptive because that metric was added after the run, which means it has no experiment interval here. Whether that observed difference is worth it depends on what a failed task costs you, and that is a number only you have. Retries do not close the gap. Of the tasks that failed once, Haiku recovered 10 of 32 and Sonnet 6 of 20, about 30% each, against first-attempt success rates of 62% and 76%. The second try is worse than a fresh draw, as Step 1 predicted.

What this means for your failure rate

The rule is short. Dropping failures skews a latency comparison at any failure rate, and past a point it removes the percentile altogether. If a share f of your tasks fail for good, only the percentiles below 1 minus f of time to successful completion exist. Fail more than 10% and your p90 of time to successful completion is gone. At exactly 10%, the result depends on the quantile convention. This repository’s type-7 definition returns infinity. Fail more than 20% and your p80 goes with it. A duration metric alone cannot tell you which side of that line you are on, because failed tasks send no duration. The success-rate metric beside it can. So measure the failure rate before you compare latency:
  1. Define done and record it. Send task-resolved only when a task is verifiably complete, not when the provider returns. In LaunchDarkly it becomes a conversion metric that counts every assigned request. Failures stay in the comparison.
  2. Put success, latency, and cost on one experiment. Read success rate first, with p90 among successes beside it and cost from the tokens the AI SDK already reports. Let the experiment quantify how uncertain each difference is.
  3. Choose on success within a time budget, next to cost. That pair answers the question you actually have: which variation completes the most tasks successfully, quickly, for the money.

Ship the decision with a guarded rollout

If the experiment points to a variation, shipping it is a config change. Stop the iteration and set the config’s default rule to serve that variation. Every application that reads the config picks up the new model on its next evaluation, with no redeploy. A guarded rollout protects that switch. It serves the new variation to a growing share of traffic, watches the metrics you choose, and can roll back automatically when one regresses. Use task-resolution as the guardrail. A model or prompt change that starts failing more tasks then rolls back before it reaches most users, even if its p90 among successes looks better. A guarded rollout and an experiment do not run on the same config at the same time. Start the rollout after you stop the iteration. Guarded rollouts are available on select plans.

Run this on your traffic

Put your requests in place of data/bfcl_confirmatory.jsonl, or send live traffic through the same config. The experiment works unchanged on production requests: point evaluate_with_bfcl in llm_p90/scoring.py at your own definition of done and keep the track() calls. To design the same kind of experiment for your own application, the launchdarkly-experiment-guide skill reads your codebase and LaunchDarkly account, proposes a hypothesis and a metric, and drafts the experiment for you to review. A later prompt change that drops resolution from 98% to 83% will look like a latency win on p90 among successes if you only chart durations. The resolution metric, on an experiment or as a guarded-rollout guardrail, is what stops you from shipping that. Part 2 picks up where this leaves off. Haiku is the cheaper model per on-time success, but it fails one task in four, and every request resends all 128 tool definitions. Anthropic’s tool search loads tool definitions on demand instead of sending all of them on every call. Can it make the cheaper model complete more tasks successfully, faster, for less? Part 2 holds the model fixed and runs that experiment.