AI Agent Tracing
A trace is one run of your agent, cut into timed pieces you can read at a glance. It is the single most useful thing you can build, because agent bugs are structural: the wrong tool, the extra step, the retry that hid a failure. This page is mostly a waterfall you can click.
A trace is a run, cut into spans
One trace is one run of your agent. Inside it, one span is one operation that took time: a model call, a tool call, a retrieval, a check. Spans nest, they carry attributes, and they have a status. That is the entire data model, and everything a tracing vendor sells you sits on top of it.
| Term | Means | For an agent |
|---|---|---|
| Trace | The whole request, with one id. | One run, from goal to answer, including every retry. |
| Span | One timed operation inside it. | One model call, one tool call, one guardrail check. |
| Parent | The span that caused this one. | Sub-agents nest under the orchestrator span, which is how you read a team. |
| Attributes | Key and value pairs on a span. | Model, token counts, tool name, arguments, result size, error type. |
| Status | ok or error, plus a message. | Set it on the span that actually failed, not only on the root. |
Live: read a waterfall
Three recorded traces from an agent doing the same job, and one you can record yourself. Click any span to see its attributes. The skill this page is teaching is a small one and it never stops being useful: look at the bars, find the widest, and ask why.
Optional: load a real model
Recording a real trace works in either mode. With a model loaded the model spans are real generations and the waterfall gets a lot more interesting, because generation is almost always the widest bar.
Live: sampling, or how not to store everything
Full traces on every run get expensive fast. Sampling is how teams cope, and the rule is always the same: drop boring successes, never drop a failure. Move the slider and watch what survives.
Attribute names worth copying
OpenTelemetry has conventions for AI spans. Use them even if you never send a byte to a vendor, because the names are the thing your future dashboards and your future tooling will expect.
| Attribute | Goes on | Example |
|---|---|---|
gen_ai.operation.name | model span | chat, generate_content, execute_tool |
gen_ai.request.model | model span | the exact model id, not the family |
gen_ai.usage.input_tokens | model span | 1840 |
gen_ai.usage.output_tokens | model span | 210 |
gen_ai.tool.name | tool span | lookup_order |
error.type | any failed span | timeout, rate_limit, validation |
your own: prompt.version, agent.step, run.id | every span | the three you will miss most if you skip them |
Three things to try, in this order
1. Open the slow trace and find the widest bar
It is not the model. It is retrieval, and it is one call. Nearly every "the agent is slow" complaint ends at a single wide bar, which is why the waterfall is the first thing to open and the dashboard is the second.
2. Open the retry trace
Three identical tool spans in a row, each failing, then a model call that answers anyway. The run status is ok. Only the span statuses show that anything went wrong, which is why you set status on the span and not just the root.
3. Set sampling to 5% and untick "always keep errors"
Watch the failures disappear from the kept set. That is the configuration a team ships when it is trying to cut its observability bill, and it is the configuration that makes the next incident unexplainable.
How tracing fails
One span per run
A trace with a single span is a log line with extra steps. The value is entirely in the breakdown.
No token or cost attributes
Then you can see that a run was slow, and never that it was expensive. For agents those are different failures.
Broken context across async work
Parallel tool calls losing the parent span produce orphan traces, which is the most common instrumentation bug in agent code.
Sampling that hides the tail
Head sampling at a flat rate is cheap and drops exactly the runs you needed. Keep every error and every slow run regardless of rate.
Related: Observability · Logging · Metrics · Monitoring · Orchestrator agents · all patterns
Free from AI School - no signup, everything runs in your browser.