Computer-use agents
When a system has no API, an agent can use it the way a person does: read the screen, move the mouse, type. It works, it is slow, it breaks when a button moves, and it needs the strictest permission design of anything in this track.
How it works
- Confirm there is genuinely no API. An API is faster, cheaper, and testable.
- Scope the agent to specific sites or applications, never the whole machine.
- Log a screenshot or the accessibility tree at every step, because this is the least debuggable agent type there is.
- Stop before every irreversible click: send, pay, delete, publish.
- Treat everything on screen as untrusted input, because a page can contain instructions aimed at your agent.
Optional: load a real model
See it work
When it pays, and when it does not
| Use it when | Skip it when |
|---|---|
| The system has no API and never will | An API exists, in which case use it |
| The workflow is stable and visual | The UI changes weekly, where the agent will break constantly |
| Volume is low to moderate | You need thousands per hour, where UI driving is too slow |
| A human can supervise the rollout | It must run unattended from day one |
How it fails
Layout changes
A moved button breaks everything. Anchor on accessibility labels and text, not coordinates, and expect maintenance.
Prompt injection from the page
The page is untrusted content the agent reads as input. Every rule from the guardrails page applies double here.
Silent wrong clicks
Without screenshots per step, you will never work out what it actually did.
What it costs
| Speed | Seconds per interaction. Orders of magnitude slower than an API call. |
|---|---|
| Maintenance | The highest of any agent type, because you depend on someone else's interface. |
| Risk | Highest too. This agent can do anything a logged-in human can do, which is why scope is everything. |
Use it as a bridge, not a foundation. Computer use is how you automate the system nobody will ever build an API for, and it should always feel like a temporary answer.
Related: Guardrails · Human in the loop · Tool calling · all patterns · agent jobs
Related: Guardrails · Human in the loop · Tool calling · all patterns · agent jobs
Free from AI School - no signup, everything runs in your browser.