Measuring an AI agent requires a second set of KPIs, separate from the business metrics you already track, because a domain KPI reports the joint output of the agent, the humans correcting it, and an environment that moved on its own — and cannot tell you which of the three did the work. The ten parameters below are the set I would instrument, drawn from production practice, academic evaluation research, and the governance frameworks that are starting to require them.

Why your business KPIs cannot see your agents

The clearest statement of this problem I have found in production comes from a telecom operator, not a bank. Deutsche Telekom put a three-agent system called RAN Guardian into live operation for radio network optimization, and described its construction to Omdia analyst Roz Roseboro in mid-2026 (Omdia research note, subscription required). An event agent gathers signals about upcoming events from news and social sources and scores its confidence in them. A monitoring agent re-frames network alarms against that event picture. A remediation agent changes network configuration as conditions build.

The business numbers are strong. Tuning that took about an hour per site now takes about a minute. A human team had been able to cover roughly a thousand events a year across Germany; once the event agent was running, it surfaced on the order of forty thousand.

But the operational detail that matters most is smaller and easier to miss. Deutsche Telekom found it needed to track two distinct sets of KPIs: the network performance measures it already had, such as throughput and latency, and a second set specific to the performance of the agents themselves — including response time and accuracy — monitored continuously to catch drift.

Here is why that second set is not optional. Throughput is a property of the network, not of the agent. A network can look healthy because the agent made a good call. It can look healthy because the agent made a bad call and an engineer caught it. It can look healthy because the agent did nothing and the week was quiet. Those three states are indistinguishable in the domain metric, and only one of them is evidence that autonomy has been earned.

Deutsche Telekom also reports that moving from proof of concept to implementation took four to five times the effort of the proof of concept itself. In my experience a meaningful share of that multiple is the measurement layer. A pilot borrows the judgment of the engineer watching it. Production has to instrument what that engineer was doing by eye, and none of it ships with the model.

The five families of agent measurement

Agent measurement converges on five families of question, and the families matter more than any individual metric, because a program that instruments four of them has a specific and predictable blind spot rather than a general shortfall.

1. Outcome — did the agent achieve the goal?
2. Trajectory — did it get there in a sound way?
3. Cost — what did the achieved outcome actually cost?
4. Safety and governance — did it stay inside the lines it was given?
5. Human interaction — what did it require from the people around it?

Measure only outcomes and you cannot separate a correct answer reached by luck from one reached by competence. Measure only trajectory and you optimize elegance over correctness. Measure only cost and you have a token budget, not a performance record. Measure only safety and the program never demonstrates value. Measure only the human load and you learn how tired your operators are.

The ten parameters, on one page

I keep the list to ten deliberately. A measurement set longer than a page does not get read at the review it exists to inform.

Parameter	Family	What it answers	Indicator type
1. Task success rate	Outcome	Did the end state change correctly?	Lagging
2. Reliability under repetition	Outcome	Does it succeed every time, not just once?	Leading
3. Tool-call accuracy	Trajectory	Right tool, right arguments?	Leading
4. Plan quality and step efficiency	Trajectory	How many steps versus how many needed?	Diagnostic
5. Groundedness	Trajectory	Is the output supported by the sources retrieved?	Leading
6. Cost per successful outcome	Cost	What did one completed unit of work cost?	Lagging
7. Boundary-violation rate	Safety	Did it try to act outside its envelope?	Leading
8. Escalation correctness	Safety	Did it hand off at the right moments?	Leading
9. Drift against baseline	Safety	Has any of the above moved since deployment?	Leading
10. Override rate and autonomy ratio	Human	How often did a human disagree, and how much ran alone?	Leading

The reliability gap is the number nobody reports

If a program could instrument only one parameter, I would choose reliability under repetition rather than task success rate.

The clearest formulation comes from the τ-bench work published by Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan at Sierra (arXiv:2406.12045, June 2024; code and data on GitHub). It evaluates agents on realistic tool-using tasks and reports not only whether an agent succeeded but whether it succeeded on every one of k repeated attempts — a measure the authors write as pass^k. Their headline finding: state-of-the-art function-calling agents of that generation succeeded on under 50 percent of tasks on a single attempt, and consistency degraded sharply under repetition, with pass^8 falling below 25 percent in the retail domain.

Read that as a governance statement rather than a benchmark result. An agent with a 60 percent single-run success rate and 25 percent eight-run consistency is not a 60-percent-reliable system that occasionally stumbles. It is a system that will handle the same request correctly on Tuesday and incorrectly on Thursday, and to everyone downstream the difference will look like an inexplicable one-off.

I call the distance between those two figures the reliability gap, and it is the most commonly omitted number in agentic AI reporting — because the single-run figure is the one that makes it into the deck.

There is a second reason to distrust single-figure capability claims. METR’s time-horizon work (arXiv:2503.14499, Kwa et al., March 2025; METR write-up) measures the length of task, expressed in how long the work takes a competent human, at which a model succeeds about half the time, and finds that horizon doubling roughly every seven months since 2019. Two things follow, pointing in opposite directions. Capability is genuinely moving faster than most governance cycles. And the standard the field reports at is 50 percent success — a reasonable convention for tracking a trend, and nowhere near a threshold for executing an irreversible action on a customer account. A 50 percent horizon is a research statistic, not a deployment license.

Never report accuracy without the cost that bought it

Sayash Kapoor, Benedikt Stroebl, Arvind Narayanan and colleagues at Princeton argued in “AI Agents That Matter” (arXiv:2407.01502, 2024; later published in Transactions on Machine Learning Research) that agent evaluation suffers from a narrow focus on accuracy without attention to other metrics, and that accuracy alone cannot identify progress because it can be improved by scientifically meaningless methods — retrying being the obvious one.

That argument is about research benchmarks. It applies with more force in production, because in production the retries are not a methodological embarrassment. They are billed.

The same group’s Holistic Agent Leaderboard (arXiv:2510.11977, 2025) put the point on an empirical footing, running more than twenty-one thousand rollouts across nine models and nine benchmarks and reporting accuracy against dollar cost rather than accuracy alone. The pattern that emerges is that the most expensive configuration is frequently not the best-performing one.

So the operational rule is short: never report a success rate without the cost that bought it. An agent that reaches 90 percent by retrying five times is a different system from one that reaches 85 percent on the first attempt. They are not close, and a report that gives only the accuracy figure has concealed the distinction that matters. This is the same discipline I argue for in my work on what agentic AI actually is and in the cost-attribution material: cost per successful outcome, with failed attempts inside the denominator.

The parameters that catch what accuracy misses

Three of the ten do work that no outcome metric can do, and in regulated settings they are the ones I would build first.

Boundary-violation rate should be reported as two numbers that must never be blended. Attempted violations are a rate and a leading indicator — a rising count means the agent is being asked to do work its scoping does not cover, which is a design signal long before it is a security one. Successful violations are not a rate at all. Each one is an incident, and the target is zero. An organization reporting a successful-violation percentage has conceded the argument: a boundary exceeded at some rate was never an enforced boundary, only a discouraged one.

Escalation correctness matters more than escalation rate, which moves for opposite reasons and is close to meaningless alone. Two error types have opposite costs. Escalating what the agent could have handled erodes the business case quietly, one unnecessary handoff at a time. Proceeding on what it should have escalated erodes everything else, occasionally and expensively. There is a further distinction most dashboards collapse: an escalation the agent initiated because it recognized its own uncertainty, and an escalation a user initiated because they gave up, are recorded identically and mean opposite things. A falling escalation rate is consistent with an agent that improved and equally consistent with users who stopped trusting it enough to try.

Drift is not really an eleventh parameter. It is the time derivative of the other nine: what you observe when the same parameter, measured the same way against the same baseline, moves. Its causes are unglamorous and mostly external — a model version change, a vendor-side routing change that never appeared in a release note, an edited prompt, an upstream schema revision, or a shift in the mix of inputs arriving from the world. The mechanism is the point: you cannot detect drift without a baseline, and a baseline is not something you can reconstruct afterward. It is a measurement you either took at deployment or did not.

How every one of these gets gamed

Every parameter here can be improved without improving anything, and naming metrics without naming their failure modes builds a target rather than a control. In my experience the gaming is almost never dishonest. It is a team under pressure making a defensible-looking definitional choice.

• Task success rate improves when the success definition loosens.
• Reliability improves when you run fewer repetitions.
• Tool-call accuracy improves when you shrink the toolset to the calls the agent already handles.
• Cost per outcome improves dramatically when failed attempts leave the denominator.
• Containment and escalation figures improve when the agent deflects rather than helps.
• The autonomy ratio improves when the definition of an autonomous action widens to include trivial ones.
• Drift goes to zero for any program that resets its baseline after every model change.

Notice the common structure. Not one of those moves falsifies a number. Every one changes a definition or a denominator, and each change is individually arguable — which is exactly why these metrics survive review while the measurement quietly stops meaning anything.

The control therefore cannot be an audit of the figures, because the figures are accurate. The control is version control of the definitions. The success criterion, the outcome unit, the repetition count, the toolset, the autonomy definition and the baseline date are all artifacts under change management, altered through the same reviewed process as a prompt or a model, with the prior definition retained so a series can be read across the change.

 

Where to start

If none of this is instrumented today, I would not start with all ten. I would write down what counts as one successfully completed task, in terms a reviewer could verify from system state rather than from the agent’s own report. Then I would record success rate and cost per successful outcome as dated baselines rather than targets. That is a week of work, it is cheap now, and it is the one artifact that cannot be recovered later.

Intent in, outcomes out — but the outcome only counts if you can still prove it six months from now.

Which of these ten is missing from your agent reporting today?

Sources

 

1. Roseboro, Roz. “Deutsche Telekom’s RAN Guardian Agent Proves Agentic AI Is Not Just Hype.” Omdia, 2026. Interview with Ahmed Hafez, Senior Vice-President for Tech Strategy and Data and AI, Deutsche Telekom. Subscription required.
2. Yao, Shunyu, Noah Shinn, Pedram Razavi and Karthik Narasimhan. “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.” Sierra, arXiv:2406.12045, June 2024. Introduces the pass^k reliability metric. Code and data.
3. Kwa, Thomas, Ben West, Joel Becker et al. “Measuring AI Ability to Complete Long Tasks.” METR, arXiv:2503.14499, March 2025. METR write-up.
4. Kapoor, Sayash, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir and Arvind Narayanan. “AI Agents That Matter.” Princeton University, arXiv:2407.01502, July 2024. Published in Transactions on Machine Learning Research, 2025.
5. Kapoor, Sayash, Benedikt Stroebl, Peter Kirgis et al. “Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation.” Princeton University, arXiv:2510.11977, October 2025.

 

© Dr. Harish Kotadia, Ph.D., All Rights Reserved, 2026.

 

Dr. Harish Kotadia, Ph.D., is an Enterprise AI Architect with 20+ years of IT consulting experience serving Fortune 100 clients, specializing in agentic AI systems built on Anthropic Claude, AWS Bedrock, and Google Vertex AI.

Disclaimer: This blog post is based on recent events and news items drawn from reputed media sources and vendor websites available in the public domain and quoted above. This post is intended for educational purposes, to help the enterprise agentic AI community learn from public information on the application and use of agentic AI tools and technology in Fortune 500 companies.

Views and opinions expressed here are my own and do not represent those of any employer or client, past or present. The analysis presented is my independent interpretation of published news reports quoted above and does not constitute legal, financial, or consulting advice of any kind.