Monitoring Agents in Production
Once an agent is live, nobody is watching individual runs. What you have instead is six numbers and a handful of rules that decide when somebody gets woken up. This page is a working dashboard for an agent you can break on purpose, so you can see which number moves first for each kind of failure.
Monitoring is not tracing
| Tracing | Monitoring | |
|---|---|---|
| Unit | One run, in full detail. | All runs, as a few numbers over a window. |
| Answers | Why did this one go wrong. | Is something going wrong right now, and how much of it. |
| Read by | A person debugging, after the fact. | A rule, every minute, whether anyone is looking or not. |
| Without the other | You can explain any single failure and never notice that it is now happening to a fifth of your traffic. | You know the failure rate tripled at 14:20 and have no way to find out why. |
You need both, and they connect: the alert names the metric, the metric names the runs, and the runs open as traces. The dashboard below is built that way, so you can click an alert and end up looking at the run that caused it. Tracing and logging cover the other half.
Optional: load a real model
Live: watch an agent in production
Press start and runs begin arriving, a few every second. The numbers are computed over the last 80 runs, exactly as a real dashboard would. Then break something from the menu and watch which tile moves first.
The alert rules
Every threshold here is editable. Tighten one until it fires on a normal day and you have built yourself an alert nobody will read by Thursday.
From the alert to the run
The last runs to arrive. Click any row to open its trace, which is the whole point of the two halves being connected.
| Runs appear here once the traffic starts. |
Three things to try, in this order
1. Break a tool, then watch the order
The tool error rate moves first, the escalation rate follows a few seconds later, and the success rate moves last because the agent keeps recovering for a while. That lag is why you alert on the leading number, not the one an executive would pick.
2. Make the model slow, and watch the cost
Latency climbs and so does the cost per run, because slow usually means extra steps rather than slow calls. Cost is a correctness signal for agents, not just a finance one.
3. Run the prompt regression
Success rate barely moves. Step count and cost per run move a lot. This is the failure that ships to production and stays there for a month, because nothing is technically broken.
The six numbers worth a dashboard
Success rate
Runs that finished with an answer that passed your own checks. Not runs that returned a 200.
Escalation rate
How often it hands over to a person. Rising means it lost confidence. Falling to zero is worse: it means it stopped asking.
Tool error rate
The leading indicator. Most agent incidents are a dependency incident wearing an agent's clothes.
Steps per run, p95
The cheapest early warning there is. Prompt changes show up here days before anyone notices a quality problem.
Cost per run
Alert on it. An agent that loops has no error to report and simply spends money until someone reads the bill.
Blocked injections
Not an outage, and not zero either. A number that stays at zero usually means the detector is broken, not that nobody is trying.
How monitoring fails
Alerting on the average
The average run is fine while the slowest fifth times out. Watch p95, and page on p95.
Alert fatigue
Six alerts a day means nobody reads the seventh. If a rule cannot be acted on at 3am, it belongs on a weekly report instead.
No quality signal at all
Uptime, latency and error rate can all be perfect while the answers quietly get worse. Sample and grade real runs weekly.
Nothing links to the trace
An alert you cannot click through to a run costs an hour of searching. Put the run id in the alert.
Related: Observability · Metrics · Tracing · Logging · Testing frameworks · Rolling out an agent · Cost control · all patterns
Free from AI School - no signup, everything runs in your browser.