AI Agent Observability
An agent that fails once in fifty runs cannot be debugged from a screenshot. Observability is what you build so that the fiftieth run leaves evidence, and it comes down to three kinds of data. This page runs one agent and shows you all three coming out of it at once.
Three signals, and nothing else
Observability sounds like a product category. It is really just three kinds of data, and the only question that matters is whether you can answer a question about your agent tomorrow without adding code today.
| Signal | What it is | The question it answers | What it costs |
|---|---|---|---|
| Traces | One run broken into timed, nested spans: the model call, each tool call, the checks. | Why did this run behave like that, and where did the time and the tokens go. | Storage per run, and about a day of instrumentation. |
| Logs | The events inside a run, with their full payloads: the prompt, the arguments, the raw tool result. | What exactly did the agent see before it decided that. | The largest volume, and the real privacy surface. |
| Metrics | Counters and histograms over all runs: successes, errors, duration, tokens, cost. | Is this happening to everyone, and did it start today. | Almost nothing. Cheap to keep for years. |
Three pages go deeper: tracing for the span tree and sampling, logging for the event schema and redaction, and metrics for choosing the numbers and putting targets on them. Monitoring is what you do with all of it once it exists.
Optional: load a real model
Live: one run, three signals
This is the part that makes it click. Run one agent and watch the same run come out as a span tree, as log events, and as metric counters. Nothing here is mocked: the numbers are measured off the run you just triggered.
Trace
Run something above.
Logs
Run something above.
Metrics
Run something above.
What you can and cannot answer
Untick a signal and the questions it was carrying go red. This is the whole design exercise: decide which questions you must be able to answer, then keep only the signals those questions need.
Three things to try, in this order
1. Turn off logs, keep traces
Most of the questions stay green. The one that dies is the only one that matters during an incident: what did the tool actually return. A trace tells you the call happened and took 900ms. It does not tell you the answer was wrong.
2. Turn off the run id
Every signal still exists and none of them can be joined up. A run id in the user-facing error message is the cheapest observability feature there is, and the one teams add last.
3. Turn off redaction, then read the log column above
Look at what is in those payloads. Everything the agent saw is now in your log store, under someone else's retention policy. Redaction is not a compliance checkbox, it is the difference between a log store and an incident.
What to instrument first, in order
| 1 | A run id, generated at the start, attached to every signal and shown to the user when something fails. |
|---|---|
| 2 | One span per model call and per tool call, with duration, status, and token counts. |
| 3 | A log event per tool call carrying the arguments and the raw result, redacted at write time. |
| 4 | Four counters: runs, failures, tool errors, escalations. That is enough for a first dashboard. |
| 5 | The prompt version and model id on every run, or you cannot compare this week to last week. |
| 6 | Cost per run, derived from the token counters. Agents fail by spending, not only by erroring. |
Related: Tracing · Logging · Monitoring · the short pattern page · Testing agents · all patterns
Free from AI School - no signup, everything runs in your browser.