AI Agent Metrics

Metrics are the cheapest signal an agent produces and the easiest to get wrong, because the numbers that are simple to collect are rarely the numbers that tell you whether the thing is working. Four questions, six numbers, and a target on the two that matter. This page lets you build that set and then attacks it.

The cheap signal, and the one people get wrong

Metrics are numbers over every run, with no payloads in them. They cost almost nothing, they can be kept for years, and they are the only signal that tells you whether something is happening to everybody. The hard part is not collecting them. It is choosing them, because the easy numbers (uptime, average latency, tokens) can all look perfect while the agent quietly does its job badly.

Question	The metric that answers it	What it misses on its own
Does it work?	Task success rate, escalation rate, tool error rate.	Says nothing about whether the successful answers were any good.
Is it fast enough?	p95 run duration, steps per run, time to first token.	An agent can be fast because it gave up early.
What does it cost?	Tokens per run, dollars per run, dollars per resolved case.	Cheap runs that fail are the most expensive thing you can ship.
Is it any good?	Grounding pass rate, human edit rate, reopen rate, thumbs down rate.	Needs sampling and judgement, which is why teams skip it and then cannot tell quality drift from noise.

This page is about choosing and defining the numbers. Monitoring is what you do with them once they exist, tracing is how you explain any one of them, and observability is the map of all three.

Live: build your metric set

Tick what you would actually collect. The scorecard checks whether you have covered all four questions, and the definitions underneath are real, ready to paste. Almost every first metric set covers three of the four, and it is always the same one missing.

Live: the average is lying to you

Agent latency is never a bell curve. Most runs are quick, a few take a retry or an extra step, and those are the ones people complain about. Drag the tail up and watch the average barely move while the ninety fifth percentile doubles.

Runs with a bad tail 8% How bad 6x

Live: turn a metric into a promise

A metric with no target is a number nobody acts on. An objective plus an error budget turns it into a decision rule: how many failures you have agreed to tolerate, how fast you are spending them, and what happens when they run out.

Target success rate % Runs per day Window days Observed success rate %

Live: the label that bankrupts you

Every label multiplies. A metric with four labels is not four things, it is the product of their values, and one careless label turns a cheap signal into the most expensive thing in your stack. Tick user_id and watch.

Metrics that look responsible and are not

Uptime

The service was up for every one of the runs where the agent gave a confidently wrong answer. Uptime is a metric about your infrastructure, not about your agent.

Average latency

Hides the tail by construction. Everyone experiences a percentile, nobody experiences an average.

Number of runs

A usage number, not a value number. It goes up when the agent is failing and people retry.

Thumbs up rate

Only a few percent of people rate anything, and the ones who do are the angry ones. Useful as a direction, useless as a level.

Tokens used

Without task success next to it, this rewards the agent that answers badly and quickly.

Percent of work touched by AI

The metric an executive asks for. It measures adoption, and it goes up just as fast when the agent is making more work than it saves.

Which metric catches which failure

The failure	The number that moves first
A dependency starts timing out	Tool error rate, minutes before success rate notices.
Someone edits the prompt and it gets chatty	Steps per run and tokens per run. Success rate barely moves.
The retrieval index degrades	Grounding pass rate, then escalation rate.
The agent starts refusing valid work	Escalation rate up, cost per resolved case up, task success down.
A loop appears in the plan	Cost per run, long before anything errors.
Quality drifts after a model upgrade	Human edit rate and reopen rate. Nothing else will show it.

Three things to try, in this order

1. Untick everything in "is it any good"

The scorecard still looks respectable, and that set is exactly what most teams ship. It can tell you the agent is up, fast and cheap while it answers worse every week.

2. Put the tail at 3% and the severity at 12x

The average moves by a fraction of a second. The p95 nearly doubles. Three percent of your users are having a different product experience and the headline number cannot see them.

3. Add user_id as a label

Series count goes from hundreds to millions in one click. This is the single most common way an observability bill explodes, and it always arrives as a helpful pull request.

Six numbers, defined once, beats sixty collected by accident. Pick one metric per question, write down how it is computed and what you will do when it moves, and put a target on the two that matter. Everything else is a dashboard nobody opens.

Related: Observability · Tracing · Logging · Monitoring · Cost control · Evaluation harness · all patterns

Free from AI School - no signup, everything runs in your browser.