Rolling out an agent
The demo works. That tells you almost nothing about production, where the inputs are messier and the failures land on real people. Every agent that survives went through the same four stages, and the value is in the gates between them.
How it works
- Shadow: it runs on real inputs, its output goes nowhere, and you compare against what humans did.
- Assist: humans see the output before acting. Measure how often they change it.
- Approve: the agent does the work and a person clicks the last step. Measure approval rate and time.
- Auto: unattended, with sampling, budgets, and a rollback. Measure the same things forever.
- Never skip shadow. It is the only stage where being wrong is free.
Optional: load a real model
See it work
When it pays, and when it does not
| Use it when | Skip it when |
|---|---|
| The agent touches real work | It is a personal tool with no blast radius |
| Other people depend on the output | You are the only user and can see every failure |
| Failures are hard to notice | Failures are loud and immediate, which shortens the shadow period |
| You need to justify the investment | It cost an afternoon and nobody needs convincing |
How it fails
Skipping shadow
The stage that costs nothing and teaches the most, skipped because the demo looked good.
Promotion by fatigue
The approval queue gets long, so someone turns on auto. That is a queue problem, not a readiness signal.
No path back
Teams plan the rollout and not the rollback, and then cannot switch it off cleanly during an incident.
What it costs
| Time | Weeks, not days. Shadow mode alone should run for a fortnight of real volume. |
|---|---|
| Double running | During shadow and assist you pay for the agent and the human. That is the price of evidence. |
| Discipline | The hardest cost. Every stage has a gate, and gates are only useful if you sometimes fail them. |
Shadow mode is free evidence. Skipping it is the most common reason an agent project gets cancelled after one bad week in production.
Related: Human in the loop · Evaluation harness · Tracing · all patterns · agent jobs
Related: Human in the loop · Evaluation harness · Tracing · all patterns · agent jobs
Free from AI School - no signup, everything runs in your browser.