Proving ROI with data-driven AI agent experiments
Proving ROI with data-driven AI agent experiments
Proving ROI with data-driven AI agent experiments
This guide was published in 2025, before LaunchDarkly shipped several features that supersede or extend the experiment patterns shown below. The methodology documented in this guide still works, but for new experiments you may also want to use:
To learn more, read AgentControl.
This guide explains how to measure and prove the return on investment (ROI) of AI model changes with LaunchDarkly experiments. It shows you how to use statistical analysis to find unexpected performance wins and cost paradoxes.
Imagine a scenario where your CEO asks you “Is the new expensive AI model worth it? And does the enhanced privacy justify the cost?” By the end of this guide, you will be able to answer this question.
You will run two A/B tests that reveal:
You can use this guide in one of two ways:
If you choose to complete the full tutorial, you need the following prerequisites:
The agent costs associated with the experiment may cost anywhere from $5 to $35. The default walk-through uses Claude Opus 4 (premium model) for testing. To reduce costs while still learning the experimentation patterns, you can modify bootstrap/tutorial_3_experiment_variations.py in your cloned repository to test with the free Mistral model instead.
To reduce costs, change the following in the create_premium_model_variations function:
This reduces the experiment cost by about $20. You will still have costs from the control group using GPT-4o and other agents in the system.
Your AI system will automatically test variations on simulated users, collecting real performance data that flows directly to LaunchDarkly for statistical analysis.
Here’s how:
The two experiments can run independently. Users can participate in both, but the results are analyzed separately.
Our supervisor agent routes PII queries to the security agent, then to support, while clean queries go directly to support. LaunchDarkly tracks metrics at the user level across all agents, which can reveal system-wide effects.
This guide explains how to run two concurrent experiments:
The experiments are explained below.
Here is an overview of the first experiment:
Here is an overview of the second experiment:
This section explains how to configure the metrics needed for your two experiments.
Navigate to Metrics in LaunchDarkly and create three custom metrics:
Expand the sections below to view the full metric setups.
Here is the detailed setup for the P95 latency metric:
Here is the detailed setup for the total tokens metric:
Here is the detailed setup for the cost per request metric:
In addition to the above custom metrics, you will also use these built-in metrics:
The cost tracking is implemented in utils/cost_calculator.py, which calculates actual dollar costs using the formula (input_tokens × input_price + output_tokens × output_price) / 1M. The system has pre-configured pricing for each model as of October 2025: GPT-4o at $2.50/$10 per million tokens, Claude Opus 4 at $15/$75, and Claude Sonnet at $3/$15. When a request completes, the cost is immediately calculated and sent to LaunchDarkly as a custom event, enabling direct cost-per-user analysis in your experiments.
Create the experiment variations using this bootstrap script:
This creates the claude-opus-treatment variation for the “Premium model value analysis” experiment. To verify the script worked correctly, navigate to your “support-model-config” feature flag in LaunchDarkly. You should now see the claude-opus-treatment variation alongside your existing variations. The “Security agent analysis” experiment will use your existing baseline and enhanced variations. Both experiments use the existing other-paid configuration as their control group.
In the left navigation, expand AI, then select Configs, then open security-agent. In the right navigation, click the plus + sign next to Experiments to create a new experiment.
Here is how to set up your experiment:
userPositive feedback rate: Select first to set as PrimaryNegative feedback ratep95_total_user_latencyai_cost_per_requestIf Context > is in Segment then Other PaidBasic Security100% of users in this experimentpii-detector: 0%Basic Security: 50%Strict Security: 50%Basic Security as the control variation, otherwise you won’t be able to allocate 50% traffic to it.Basic SecurityBayesian90%Click Save, then click Start experiment.
You may see a “Health warning” indicator after starting the experiment. This is normal and expected when no variations have been exposed yet. The warning will clear after your experiment starts receiving traffic and data begins flowing.
In the left navigation, expand AI, then select Configs, then open support-agent. In the right navigation, click the plus + sign next to Experiments to create a new experiment.
Here is how to set up your experiment:
userPositive feedback rate: Select first to set as PrimaryNegative feedback ratep95_total_user_latencyaverage_total_user_tokensai_cost_per_requestIf Context > is in Segment then Other Paidother-paid100% of users in this experimentrag-search-enhanced: 0%eu-free: 0%eu-paid: 0%other-free: 0%other-paid: 50%international-standard: 0%claude-opus-treatment: 50%Basic Security as the control variation, otherwise you won’t be able to allocate 50% traffic to it.other-paidBayesian90%Click Save, then click Start experiment.
You may see a “Health warning” indicator after starting the experiment. This is normal and expected when no variations have been exposed yet. The warning will clear once your experiment starts receiving traffic and data begins flowing.
If two independent experiments are running concurrently on the same users, then each user experiences:
Basic Security or Strict Security)Claude Opus 4 Treatment or Other Paid (GPT-4o))Random assignment ensures balance, because ~50% of users receive each combination.
Start your backend and generate realistic experiment data. Choose between sequential or concurrent traffic generation.
We recommend this option for large datasets.
Here is the code for faster experiment data generation with parallel requests:
This configuration includes:
Runtime depends largely on if you retain MCP tool enablement as these take much longer to complete.
We recommend this option for simple, one-at-a-time traffic generation:
What happens during traffic simulation:
Here is an example of the traffic generation output you might see:
Relevant performance information includes:
semantic_scholar MCP tool may take 5-20 minutes due to API rate limitsRefresh your LaunchDarkly experiment Results tabs to see data flowing in. Cost metrics appear as custom events alongside feedback and token metrics.
After your experiments have collected data from about 100 users per variation, you’ll see results in the LaunchDarkly UI. Here’s how to interpret them:
Question: Does enhanced security improve safety without significantly impacting positive feedback rates?
Answer: Not all metrics reach significance at the same rate. In our security experiment we ran over 2,000 more users than the model experiment, turning off the MCP tools and using --pii-percentage 100 to maximize PII detection.
This is what we found:
This is normal. Low-variance metrics (latency, tokens) produce results quickly. High-variance metrics (cost, feedback) need massive samples. You may not be able to wait for every metric to hit 90%. Some metrics produce strong signals quickly, but for others, draw conclusions from the data and judgment based on your own experience.
These results show that enhanced privacy is worth the modest cost.
The results tell a compelling story:
Strict security uses free Mistral for PII detection, yet increases total system cost by 14.6%:
Why does the support agent cost more? More aggressive PII redaction removes context, forcing the support agent to generate longer, more detailed responses to compensate for the missing information. This demonstrates why system-level experiments matter. Optimizing one agent can inadvertently increase costs downstream.
Decision logic:
Conclusion: Deploy strict security. We expected latency to stay within 3 seconds of baseline, but discovered a 36% improvement instead (with 87% confidence). Mistral is significantly faster than Claude Haiku. Combined with enhanced privacy protection, this more than justifies the modest 14.5% cost increase (with 79% confidence).
At scale, paying ~$0.004 more per request for significantly better privacy compliance and faster responses is a clear win for EU users and privacy-conscious segments.
Here are the results:
Question: Does Claude Opus 4 justify its premium cost with superior positive feedback rates?
Answer: Reject Claude Opus 4.
The experiment delivered a decisive verdict:
Decision logic:
Conclusion: Premium price delivered worse results on every metric. Experiment was stopped when positive feedback rate reached significance.
GPT-4o dominates on performance and speed and most likely also on cost.
Here are the results:
Here’s what we learned running these experiments ourselves:
Low-variance metrics (latency, tokens) reach significance quickly (~1,000 samples). High-variance metrics (cost, feedback) may need 5,000-10,000+ samples. This isn’t a flaw in your experiment but the reality of statistical power. Don’t chase 90% confidence on every metric; focus on directional insights for high-variance metrics and statistical proof for low-variance ones.
Using a free Mistral model for security reduced that agent’s cost to $0, yet increased total system cost by 14.5 because downstream agents had to work harder with reduced context. However, the experiment also revealed an unexpected 36% latency improvement. Mistral is not just free but significantly faster. LaunchDarkly’s user-level tracking captured both effects, enabling an informed decision: enhanced privacy + faster responses for ~$0.004 more per request is a worthwhile tradeoff.
At 87% confidence for latency (as compared to 90% target), the 36% improvement is clear enough for decision-making. Perfect statistical significance is ideal, but 85-89% confidence combined with other positive signals (stable feedback, acceptable cost) can justify deployment when the effect size is large.
We encountered some limitations while running these experiments:
Model-as-judge evaluation: We use Claude to evaluate response quality rather than real users, which represents a limitation of this experimental setup. However, research shows that model-as-judge approaches correlate well with human preferences, as documented in Anthropic’s Constitutional AI paper.
Independent experiments: While random assignment naturally balances security versions across model versions, preventing systematic bias, you cannot analyze interaction effects between security and model choices. If interaction effects are important to your use case, consider running a proper factorial experiment design.
Statistical confidence: LaunchDarkly uses Bayesian statistics to calculate confidence, where 90% confidence means there’s a 90% probability the true effect is positive. This is not the same as p-value < 0.10 from frequentist tests. We set the threshold at 90%, rather than 95%, to balance false positives versus false negatives, though for mission-critical features you should consider raising the confidence threshold to 95%.
Here are common mistakes you avoided, and what you did instead:
You’ve built a data-driven optimization engine with statistical rigor through falsifiable hypotheses and clear success criteria. Your predefined success criteria ensure clear decisions and prevent post-hoc rationalization. Every feature investment now has quantified business impact for ROI justification, and you have a framework for continuous optimization through ongoing measurable experimentation.
Findings from our experiments include:
This section includes troubleshooting solutions.
If you see requests taking exceptionally long, the root cause is likely the semantic_scholar MCP tool hitting API rate limits, which causes 30-second retry delays. Queries using this tool may take 5-20 minutes to complete. The 2000-second timeout handles this gracefully, but if you need faster responses (60-120 seconds typical), consider removing semantic_scholar from tool configurations. You can verify this issue by checking logs for HTTP/1.1 429 errors indicating rate limiting.
If ai_cost_per_request events aren’t showing in LaunchDarkly, first verify that utils/cost_calculator.py has pricing configured for your models. Cost is only tracked when requests complete successfully (not on timeout or error). The system flushes cost events to LaunchDarkly immediately after each request completion. To debug, look for COST CALCULATED: and COST TRACKING (async): messages in your API logs.
In this guide you ran two experiments:
AgentControl configs support two modes: prompt‑based (single‑step completions) and agent‑based (multi‑step workflows with tools). Below are additional patterns to explore.
Experiments you can run entirely in AgentControl configs with no app redeploy include:
Prompt and template experiments (prompt‑based or agent instructions): Duplicate a variation and iterate on system/assistant messages or agent instructions to measure adherence to schema, tone, or qualitative satisfaction. Use LaunchDarkly Experimentation to tie those variations to product metrics.
Model‑parameter experiments: In a single model, vary parameters like temperature or max_tokens, and optionally add custom parameters you define, for example, an internal max_tool_calls or decoding setting, directly on the variation.
Tool‑bundle experiments (agent mode or tool‑enabled completions): Attach/detach reusable tools from the Tools Library to compare stacks, such as search_v2, a reranker, or MCP‑exposed research tools, across segments. Keep one variable at a time when possible
Cost/latency trade‑offs: Compare “slim” versus “premium” stacks by segment. Track tokens, time‑to‑first‑token, duration, and satisfaction to decide where higher spend is warranted.
Recommendations include:
Experiments you can run with feature flags include:
Fine‑grained RAG tuning: k‑values, similarity thresholds, chunking, reranker swaps, and cache policy are typically coded inside your retrieval layer. Expose these as flags or custom parameters on your AgentControl configs if you want to A/B test them.
Tool‑routing guardrails: Fallback flows (for example, retry with a different tool/model on error), escalation rules, or heuristics need logic in your agent/orchestrator. Gate those behaviors behind flags and measure with custom metrics.
Safety guardrail calibration: Moderation thresholds, red‑team prompts, and PII sensitivity levers belong in a dedicated safety service, for example, enterprise as compared to free.
Session budget enforcement: Monitoring will show token costs and usage, but enforcing per‑session or per‑org budgets (denylist, degrade model, or stop‑tooling) requires application logic. Wrap policies in flags before you experiment.
These ideas work with all of the experiment suggestions we listed above:
Require statistical evidence before shipping configuration changes. Pair each variation with clear success metrics, then A/B test prompt or tool adjustments and use confidence intervals to confirm improvements. When you introduce the new code paths above, protect them behind feature flags so you can run sequential tests, multi-armed bandits for faster convergence, or change your experiment design to understand how prompts, tools, and safety levers interact.
Across this three-part series, you’ve transformed from hardcoded AI configurations to a scientifically rigorous, data-driven optimization engine. Part 1 established your foundation with a dynamic multi-agent LangGraph system controlled by LaunchDarkly AgentControl, eliminating the need for code deployments when adjusting AI behavior. Part 2 added sophisticated targeting with geographic privacy rules, user segmentation by plan tiers, and MCP (Model Context Protocol) tool integration for real academic research capabilities. This tutorial, Part 3, completed your journey with statistical experimentation that proves ROI and guides optimization decisions with mathematical confidence rather than intuition.
You now possess a defensible AI system that adapts to changing requirements, scales across user segments, and continuously improves through measured experimentation. Your stakeholders receive concrete evidence for AI investments, your engineering team deploys features with statistical backing, and your users benefit from optimized experiences driven by real data rather than assumptions. The chaos of ad-hoc AI development has given way to clarity through systematic, scientific product development.