Module 12 · Agent types · Lesson 4 of 7
Computer-use and browser agents
Agents that see a screen or a DOM and act on it: screenshots against the accessibility tree against scripts, why each step is brittle and dear, irreversible actions and confirmation gates, and when an API is the honest answer.
Why it matters
Some work has no API. A supplier portal from 2009, an internal desktop tool, a government form: the only interface is the one a person uses. Computer-use and browser agents close that gap by looking at a screen and acting on it with a mouse and keyboard. They are also the agent type most likely to do something irreversible, because the click that reads a page looks the same as the click that submits a payment. Interviewers ask about them to see whether you know when a GUI agent is the right tool and when it is an expensive way to avoid asking for an API.
The idea
Observation
The loop is observe, decide, act, observe. Three ways to observe. Screenshots: the model receives an image and answers with a coordinate to click or text to type. General, works on anything with pixels, and each step costs an image's worth of tokens plus a vision pass. Accessibility tree or DOM: the model receives a structured list of elements with roles, names and ids, and acts by element reference. Cheaper, exact, and blind to anything drawn without semantics, such as a canvas or a custom widget. Code-driven automation: the model writes a script against a browser automation library, selectors and all, and the runtime executes it. Fastest and most repeatable once it works; brittle to layout changes and useless when the page is unknown. Production agents are usually hybrid: tree or script first, a screenshot when the tree is empty or a step fails.
Why each step is brittle and expensive
A twenty-step task is twenty images of a few thousand tokens each, read fresh every time, with no cheap way to know whether the page changed. Clicks are placed by coordinate, and a dropdown that opened, a banner that appeared or a page that scrolled moves the target. The model cannot tell a loading spinner from a finished page without waiting and looking again. Each step carries a few percent chance of a wrong click, and thirty steps compound that into failure most of the time unless recovery is designed in: re-observe after every action, compare with the expectation, undo or navigate back when it differs.
Irreversible actions and confirmation gates
Reading is safe; submit, pay, send and delete are not, and the agent cannot always tell which button is which. The runtime classifies actions: by element text and role, by URL patterns, by a policy the developer wrote. Before any action in the irreversible class the agent stops and asks, showing the screen it is about to act on. On the user's side this is a button, not a chat message the model interprets. It is the only reliable brake, because the model's judgement about what a button does is exactly what fails.
Sandboxed profiles
The agent runs in a browser profile of its own: fresh cookies, no saved passwords, no payment methods, an allow-list of domains, and the user's session handed over only for the site the task needs. Injected text on any page can tell the agent to go elsewhere; the profile limits what elsewhere can reach.
When it is the right tool
A browser agent is right when no API exists, when the GUI is legacy and will not get one, and when the task is testing the UI itself. It is the wrong tool when an API or an MCP server exists or could exist after a day's work: the structured path is faster, cheaper and does not misclick. The honest answer in a design review is often "wrap the system in a tool server and keep the browser for the one page that cannot be wrapped".
| Screenshot and coordinates | DOM or accessibility tree | Code-driven automation | |
|---|---|---|---|
| Reliability | Lowest: coordinates drift with layout, banners and scroll | High where elements have semantics; blind to canvases and custom widgets | High on a known page; breaks when the page changes |
| Cost per step | Highest: an image plus a vision pass every action | Low: text tokens for the tree | Lowest: one script runs many steps with no model call between |
| Generality | Anything with pixels, including desktop apps | Web and apps that expose a tree | Pages you have seen before |
| Speed | Seconds per step | Under a second per step | Milliseconds per step once written |