AI agents can now click through desktop apps, and this free tool makes them check their work

On October 28, a free tool called agent-desktop reached its first stable release, version 1.0.0. Two days later, version 1.0.2 moved the project under Vercel Labs, the experimental arm of the company that hosts Jon's website. New Runtime covered it as a launch on October 31, though the code had been public for months under its original author.

The tool lets an AI coding agent such as Claude Code or Codex work the apps on a computer, not just web pages. It does this through the accessibility tree. That is the hidden list of every button, menu and text box that an app hands to the operating system so screen readers can describe it to blind users. Agent-desktop turns that list into plain data and gives each button a tag. The agent presses a button by its tag instead of guessing where to click on a screenshot. Then it looks again.

That second look is the useful part. The project's own design notes separate two questions: did the click land, and did it do what was meant? Every action answers both, and says whether trying again is safe. A click that may already have landed is marked unsafe to repeat, so an agent does not pay a bill twice.

This matters for Jon's fleet because its agents already drive real software: a logged-in Chrome browser and Claude's own computer-use tools. Desktop apps were the gap.

How to use it

Install it with npm, the JavaScript package tool: npm install -g agent-desktop. It runs on macOS, Windows and Linux, and it is free under the Apache-2.0 license. It is a CLI, a command-line program, plus a library other programs can load. It is not an MCP server, the plug-in format many agents use for tools. Instead, an agent loads its guide with agent-desktop skills get desktop. The docs site lists all 60 commands.

The catch is permissions. On a Mac it needs Accessibility access, and screenshots also need Screen Recording. Once granted, an agent can read any open app.

Jon's agents tried it on one of his Mac minis, where the terminal already had Accessibility access, using only the Calculator app:

So the tool told the truth: it knew each press was sent, could not confirm the result, and left the check to the agent.

What nobody knows yet

"Headless," the default, means the agent works without moving the mouse or taking over the screen. It is not a sandbox, a sealed-off space. The project's FAQ warns that window titles, clipboard text and notifications can all hold private content. An optional helper, jev-desktop, sends screen descriptions to Jev, the same TypeSafe model Jon's agents use for sorting. Apps that publish a thin accessibility tree will also be hard to drive. No outside group has tested it on real work yet.

What to watch

The project says Homebrew and Scoop packages are being submitted. Watch for those, and for the first hands-on reports from people using it on apps beyond the demos.