agent-desktop on GitHub: what it does and how to use it with AI agents

What agent-desktop does

agent-desktop is a command line tool (plus a C library) from Vercel Labs that gives an agent computer use on macOS, Windows and Linux. A snapshot returns JSON listing the buttons, fields and menus in an app, each with a ref. The agent acts on a ref, for example a click, then takes a new snapshot to check the result. There is no pixel matching, and actions are headless by default, so it does not move your pointer or steal focus. Version 1.0 shipped on Oct 28, 2026.

Who it is for

People who want Claude Code, Codex or another coding agent to operate native apps (Finder, Slack, settings panels) that have no API or MCP server.

How to use agent-desktop with Claude Code, Codex or Cursor

1. Install it: npm install -g agent-desktop (Bun, Yarn and pnpm also work), then check it with agent-desktop version.
2. On macOS, grant Accessibility permission: agent-desktop permissions --guide. On Linux it needs your graphical session with at-spi2-core.
3. Try the loop yourself: agent-desktop snapshot --app Finder -i, then agent-desktop click with a ref from that snapshot, then snapshot again to verify.
4. Give it to your agent: npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop

Tip: If your agent does not support skills, tell it to run agent-desktop skills get desktop and agent-desktop skills get platform and follow what they print. Snapshots and screenshots can include private desktop content, so keep them out of public logs.

Related repos

• Browser Use: Open-source Python library for AI agents that use a web browser.
• Playwright MCP: Microsoft's MCP server that lets an agent drive a real browser.

agent-desktop FAQ