I recently had the opportunity to speak at TechFuse DSM to a full room of engineers, product owners, and technology leaders about measuring the ROI of AI initiatives. Most teams focus on metrics like cost per developer, usage, latency, and token spend, but they miss the most important one: impact.
During the talk, I broke AI measurement down into five categories. The first four (performance, quality, behavior, and safety) tell you whether the system is running well. The fifth, impact, tells you whether it was worth building. Teams usually have a handle on the first four and skip the fifth, which is why questions about ROI are so hard to answer.
To keep this concrete, I’ll use the three example systems: a checkout agent at a grocery store that suggests broth and carrots when you put chicken and noodles in your cart, a maintenance agent in a plant that reads sensor data and flags a part before it fails, and coding agents, which I think are the most expensive AI initiative to get wrong.
Even a healthy agent can lose money
You can have an agent in production that’s fast, accurate, well-behaved, and safe, and it can still be a bad investment. Most agent dashboards show the model, tokens, latency, cost, and an eval score. Those numbers tell you the request ran fine. They can’t tell you whether the agent moved anything for the business, because nobody tied it to a business metric before it shipped.
Whether an agent is worth it comes down to one equation: Value = impact − cost
Impact is how much a business metric moved, in dollars. For AI, cost is generally tokens plus the people time it takes to run the agent, which most teams forget to include. The math is simple, but it belongs in every conversation about AI.
I’d also argue that the business metric has to exist before you apply AI to the problem. If you have to invent a metric to prove the AI worked, you might be measuring the wrong thing.
For the checkout agent, the metric is gross profit per checkout. Say it’s $14.00 before the agent and $14.25 with it. Across 2 million checkouts a month, that’s $500,000 of impact. Call it $100,000 a month to run, and you get $400,000 in value.
For the maintenance agent, the metric is unplanned downtime hours. Suppose they drop from 320 a year to 240. Siemens puts the cost of an hour of unplanned downtime in fast-moving consumer goods, the sector food plants fall under, at about $36,000. That’s roughly $2.9 million of impact against about $200,000 to run.
This comparison changes the token spend discussion. A $200,000 agent bill is hard to justify on its own. Next to $2.9 million of impact, it’s an investment with a clear ROI.
Self-reporting won’t cut it
Most teams measure coding agents by asking developers how much time they saved. Model Evaluation and Threat Research (METR) tested that assumption in early 2025 with a randomized controlled trial of 16 experienced open source developers working on 246 real tasks in their own repositories.
Going in, the developers expected AI to make them 24% faster. Afterward, they estimated it had made them 20% faster. Measured, they were 19% slower.
If developers can misjudge their own speed by that much, self-reported time savings make a shaky basis for judging a coding agent. Track a business metric like lead time for changes instead.
If the value isn’t obvious, it probably shouldn’t use AI
My hot take is that not everything should use AI, and part of the job is being willing to say, “This doesn’t need to be agentic.”
Take an agent that answers “Where’s my order?” in chat. Order status is a lookup: one table, one field, done in milliseconds. Wrapping it in an LLM adds latency, a token cost per question, and a new way to get the answer wrong, like making up a delivery date. None of that buys the customer anything a status page didn’t already give them.
The value test is to ask what AI adds here that you can’t do without it. If you can’t answer that in a sentence, it probably shouldn’t be agentic.
4 questions to ask before your next AI initiative ships
If your team tracks tokens and latency on an AI product, you already have a starting point for the value conversation. These questions will help you finish it.
- What business metric is this supposed to move? “Developer productivity” and “customer experience” are categories. Keep going until you get a number with a unit.
- What does that metric look like today, and where does it live? If the answer is “We’d have to pull that,” the baseline needs to exist before the agent ships. You can’t get it back afterward.
- Who owns that number? This tells you who’s actually invested. If no one on the business side owns it, the value conversation hasn’t happened yet.
- When you review it, what will you do with the results? If there’s no review date and no result that would make you change course, there’s no plan to find out whether it worked.
Most teams have a dashboard (or seven) full of AI metrics. My argument is that almost none of them connect those metrics to something the business cares about.
Try it yourself
To work through this for your own AI initiative, start with this worksheet and prompt that I put together for my talk.
