Vercel Labs launches agent-desktop: computer use for any coding agent via OS accessibility trees, not pixels, on macOS, Windows and Linux
On October 31, Chris Tate of Vercel Labs, creator of agent-browser, announced agent-desktop on X as "agent-browser, for your whole computer". It is an Apache-2.0 Rust CLI and C library that reads a desktop app's OS accessibility tree and returns every actionable element as a JSON ref; the agent clicks or types on a ref and snapshots again to verify, with no pixel matching, headless by default. Version 1.0.0 shipped on October 27 and added Linux support; the current npm release is 1.0.2.
Key Takeaways
- Install:
npm install -g agent-desktop, one package with native x64/ARM64 binaries; 1.0.0 (Oct 27) added Linux (glibc 2.35+, via AT-SPI), and the latest npm release is 1.0.2 (Oct 30) - Loop:
snapshotprints a JSON tree with refs,click @<snapshot_id>:eNacts, thensnapshotagain to verify; a changed UI returnsSTALE_REF, and errors say whether a retry is safe - Scope: the README lists 60 command names, 56 of them operational, spanning windows, keyboard and mouse, notifications, clipboard, waits and session traces
- Headless by default: ref actions go through accessibility APIs without moving the pointer or stealing focus, with
--headedfor physical input; Chris Tate says multiple agents can use the computer in parallel, each with its own cursor - Benchmarks: no official success-rate or speed numbers; it is a tool with no model inside, and web pages stay with agent-browser
Linked Models & Agents: