Say you have an agent that reviews refund requests and returns “approve” or “deny.” You run the same request through it twice, with the same instructions and the same model, and the first run approves it while the second denies it.
I worked in data science before I joined LaunchDarkly, and my instinct when I first hit this type of thing was to treat it as noise that averaging would wash out. But the customer waiting on the refund does not get an average. They get one answer.
You can pay for several samples and take a vote. That makes each decision more expensive, and it still leaves a question: When you change the prompt or the model, which refund requests does the agent start judging more reliably, and which does it start getting wrong?
The usual way to answer a question like that before shipping is an eval. Collect a few hundred past refund requests, one per row, and write down the decision each one should have received. Run the agent across the set and score every row against its expected decision. The share it gets right is the accuracy of that run, and one pass through the set is one run.
But accuracy alone will not answer the question, because it collapses the whole set into a single number. Instead, judge every request in the set more than once, three times to start, and report three numbers side by side:
- Mean accuracy is the share of all those judgments that were right, counting every request and every attempt.
- pass^k is the share of requests the agent got right on all k attempts. A refund it approves twice and denies once doesn’t count.
- Flip rate is the share of requests whose k judgments did not all agree with each other, right or wrong.
Each number hides something the other two show. This post covers why the usual fix, pinning temperature to zero, will not spare you from measuring consistency across repeated runs. It then discusses how to compare a candidate against the version you run today row by row, and finally, how to turn that comparison into a release decision.
Temperature zero does not stop the flip
The fix most teams reach for first is to pin temperature at zero and send every request identically. If identical inputs produced identical outputs, one run would tell you everything that k runs would, and repeat runs would be money spent on nothing. Hosted models are not always deterministic, though, and some no longer expose a temperature setting at all. Atil et al. ran five LLMs, configured to be deterministic, on eight tasks, 10 times each. They saw "accuracy variations up to 15% across naturally occurring runs with a gap of best possible performance to worst possible performance up to 70%." None of the models delivered repeatable accuracy across all eight tasks, let alone identical outputs.
Horace He and Thinking Machines Lab traced the leftover jitter to the server: The same request can get a different answer depending on how many requests were processed alongside it.
A 90%-accurate agent gets eight right in a row about 43% of the time
Suppose the agent judges any single refund correctly with probability p, independently each time. Then the chance it gets the same refund right k times in a row is p^k. At 90% per judgment, eight in a row is 0.9^8, or about 43%, even though accuracy on any single run still reads 90%. Real agents are less uniform than that: Most rows stay consistently right or consistently wrong, and the flips concentrate on hard or ambiguous inputs. pass^k measures the share of rows that survived all k attempts without flipping.
τ-bench introduced pass^k to expose that decay: On its retail set, GPT-4o with function calling went from 61.2% on one attempt to less than 25% on eight. IBM Research reported the same gap on AppWorld, where a GPT-4.1 ReAct agent averaged 77.4% across five runs while passing all five runs on only 53.0% of tasks.
Evan Miller's Anthropic paper, Adding Error Bars to Evals, also warns against turning temperature down just to make an eval look calmer. Lowering temperature can shift variance into the question-level means or bias the score, so measure the agent at the settings you ship.
The mean can hide a regression
An unstable decision doesn’t stay a statistics problem for long. If asking again can change the answer, people learn to ask again. Whoever is handling the refund resubmits it until it comes back approved, and the agent becomes a rubber stamp.
A prompt change is almost never a uniform lift. You pick up some requests that used to be misjudged, and you lose some that used to come back right every time. Mean accuracy only reports the net. If a new prompt fixes 20 requests and breaks 12 that were reliably right before, the score still goes up. Those 12 are the decisions someone was already trusting.
pass^k can hide that trade-off, too. More rows can become consistently correct than stop being consistently correct. The aggregate improves, but you still need to know which rows got worse.
And a low flip rate is not enough. Three unanimous approvals on a refund that should’ve been denied are consistent and wrong. That’s why the eval needs to distinguish stable-correct, stable-wrong, and flipping decisions, rather than only reporting how often the answers agree.
How to measure a change row by row
These steps compare a candidate prompt or model against the version you run today.
Step 1: Label the rows and mark what must not break
Start with a labeled dataset where every row has the expected decision. Before you look at any results, tag the rows that must not break, such as high-value refunds the current version correctly denies. Deciding this up front keeps you from moving the line after you see the numbers.
Step 2: Run the current version three times on every row
Keep every decision, and score approve and deny by exact match against the expected label. There’s no need to ask another LLM whether those answers agree, which would add another source of variation to the measurement.
On a 200-row dataset, three attempts per row means 600 agent executions per version. The cheaper alternative is to run each version once and repeat only the rows where their answers disagree. That helps investigate differences you’ve already found, but it doesn’t measure stability across the dataset. Both versions can happen to return the same answer once even though one changes its answer on the next attempt. So the repeat runs go on every row, and the extra confirmation runs in Step 7 go on suspected regressions.
Step 3: Save each intermediate check, not just the verdict
If the agent reaches its decision through smaller checks, such as the return window, proof of purchase, and damage criteria, save each check's answer, too. The final decision can stay the same even when an intermediate answer changes, and looking only at the verdict hides that variation.
Step 4: Classify every row and save the baseline
A row is stable-correct when all three decisions match the expected answer, stable-wrong when all three agree on the wrong answer, and a flip when the decisions differ. These describe the runs you observed, not a guarantee about the next one.
Save these results as your baseline. You can reuse them while the baseline configuration, dataset, and evaluation conditions stay the same.
Step 5: Report the three numbers together
Put mean accuracy, pass^3, and flip rate side by side. None of them goes on the report alone.
Step 6: Run the candidate and list what moved
Run the candidate three times on the same rows and classify them the same way. Then write down two lists: rows that became stable-correct, and rows that were stable-correct in the baseline and no longer are. Keep that second list even when every headline metric improves.
Step 7: Confirm each suspected regression
For every row on the second list, run both versions again on that input. Separate cases where the baseline also varies from cases where the baseline keeps getting the answer right and the candidate keeps getting it wrong. Spend more attempts on the rows you tagged in Step 1.
For each confirmed regression, compare the intermediate checks in order and find the first point where the candidate's answers diverge from the baseline. That gives you a specific prompt or rule to inspect, rather than a lower score and a guess. A single broad "Should we refund this?" instruction gives you little to investigate when the answer flips.
Turning results into a release decision
The evaluation above gives you three numbers and two lists. Repeat runs separate a real behavior change from variation that was already in the baseline, and the lists from Steps 6 and 7 name the exact rows that moved.
Start a guarded rollout only if both of these hold:
- Every must-not-break row from Step 1 that was stable-correct on the baseline is still stable-correct on the candidate after the confirmation runs in Step 7.
- Every must-not-break row has enough repeats to classify. A row you can’t classify isn’t a pass.
Compute both checks from the recorded verdicts by exact match. Step 7 is the last time you run the agent; rerunning to get a different gate result is not allowed. If either check fails, the candidate does not roll out. A higher pass^k on the rest of the set doesn’t override them.
The version ships when the guarded rollout completes without a regression on the production metrics you attached: duration, generation errors, token usage, and any satisfaction or business signal you already collect. With automatic rollback on, LaunchDarkly reverts the release if it detects one. Add any confirmed production failure to the dataset as a must-not-break row; after a version is stable-correct on it, no later candidate ships if it breaks it.
Where AgentControl fits in
LaunchDarkly AgentControl datasets hold the inputs fixed. Each row carries the input, the expected decision, and metadata such as a risk tag, and every offline evaluation run pins the dataset version it started on. This means that editing the dataset later doesn’t change a result you already have. Each run generates one output per row, so k means k runs of the same evaluation against the same dataset version, lined up by row. Metadata is stored alongside the results without affecting built-in scoring (though custom scorers in evaluations from code can read row.metadata if you want to weight or filter by tag). Combining repeated runs into row-level stability classes, comparing versions, and enforcing the release rules above happen in your own script. The offline evaluations tutorial shows how to save a reference run and compare the next change against it, and guarded rollouts handle the production release step.
Conclusion
A single eval score can climb while decisions your users already rely on start flipping. Running every labeled row multiple times, blocking release when any must-not-break row loses stability, and shipping the candidate behind a guarded rollout helps catch those regressions before customers find them by resubmitting requests.
