Human in the loop
Everyone agrees there should be a human in the loop. Almost nobody designs the loop, so the human gets a stream of approvals with no context and starts clicking yes, which is worse than no human at all because now the failure has a signature on it.
How it works
- Decide what is irreversible. Those steps always stop.
- Show the human what they need to judge: the action, the evidence, and the reason.
- Make rejecting as easy as approving, and record why.
- Watch the approval rate. Above 95% means people have stopped reading.
- Move steps out of the loop only when the data says they are safe, not when the queue gets long.
Optional: load a real model
See it work
When it pays, and when it does not
| Use it when | Skip it when |
|---|---|
| The action cannot be undone | The action is trivially reversible and cheap |
| Volume is low enough to read each one | Volume is so high that approval becomes reflex, where sampling is honest and stopping is not |
| The human can judge in under a minute | Judging properly needs research the human will not do |
| You can show the evidence next to the action | All you can show is a confidence score, which nobody can act on |
How it fails
Rubber stamping
The most common failure and the hardest to see. Track approval rate and time spent per approval.
Approval without evidence
Approve this refund with no order history means the human is guessing more than the agent was.
Queue as a bottleneck
If approvals pile up, people bulk approve. Design for the volume you actually have.
What it costs
| Throughput | The loop is the slowest step by definition. That is the trade you are making, so make it deliberately. |
|---|---|
| Human attention | The scarcest resource in the system. Spend it on the irreversible and cheap-to-judge cases. |
| Design effort | Most of the work is in the screen, not the agent. |
If approval rate is 99%, you do not have a human in the loop. You have a person who has learned that clicking approve is always right, which is a worse position than automating openly.
Related: Guardrails · Budgets and limits · Rolling out an agent · all patterns · agent jobs
Related: Guardrails · Budgets and limits · Rolling out an agent · all patterns · agent jobs
Free from AI School - no signup, everything runs in your browser.