First-draft replies
The value here is not the writing, it is the checking. Any model can produce a plausible support reply. The agent worth having is the one that refuses to hand you a draft that promises a refund, invents a date, or forgets to say what happens next.
The job on one screen
| Runs when | A ticket lands in a queue with a known intent. |
|---|---|
| Reads | The ticket, the customer's plan and history, and the help centre articles the retriever found. |
| Decides | What to say, what it is not allowed to promise, and whether the draft is good enough to show a person. |
| Produces | A draft reply in the ticket, plus the articles it used so the agent can check them. |
| Stops when | Draft written and all checks passed. Sending is always a human click. |
| Tools it needs | Knowledge base retrieval, ticketing API (draft only), customer record lookup. |
| Autonomy to start at | Approve. It does the work and stops before the irreversible step. |
Optional: load a real model
Run it
The checks the draft has to pass
The model writes, then code checks. A draft that fails any check never reaches the send step, and the failure is shown rather than silently patched.
| Check | What it protects against |
|---|---|
| No unbacked money promise | An agent that says the word refunded before finance has approved it creates a chargeback and a complaint. |
| Says what happens next | A reply with no next step generates a second ticket from the same person, which doubles the cost of the first one. |
| Under 130 words | Long support replies get skimmed, and the important sentence is the one that gets skipped. |
| No invented specific date | Models produce confident dates from nothing. If the notes had no date, the draft must not have one either. |
Where this one goes wrong
Confident invention
The model fills a gap with a plausible policy that does not exist. Ground every claim in a retrieved article, and check the draft for claims that no article supports.
Tone matching gone wrong
Mirroring an angry customer produces an angry reply. Fix the tone in the system prompt, not in the model's mood.
The stale article
A correct answer from a help page nobody updated since the pricing change. Retrieval quality is the ceiling on reply quality.
How you would know it is working
| Measure | Why that one |
|---|---|
| Edit distance before send | How much humans change the draft. Falling edit distance is the only honest sign it is getting better. |
| Reopen rate | Replies that produced a second ticket. A fast wrong answer is worse than a slow right one. |
| Check failure rate | How often the agent held its own draft back. If this is zero, your checks are decorative. |
Earning more rope
| Assist | Draft into a private note that only the agent sees. Compare against what the human actually sent. |
| Approve | Draft into the reply box, human edits and sends. Most teams should stay here indefinitely. |
| Auto | Auto-send only for a narrow, boring intent list with a rollback, such as password reset instructions, and never for anything involving money. |
Related: Ticket triage · Guardrails · Knowledge base gaps · all agent jobs · Agent Lab home
Free from AI School - no signup, everything runs in your browser.