Computer Use

computer lets an expert operate your desktop directly: native applications, games, any interface without a DOM. It is the lowest-level interaction tool available, and the one where understanding the constraints matters most.

Activation

profy-computer is user_selectable: true with requires: { desktop_connected: true }. Both conditions are mandatory — you must enable it in the plugin panel and be running Profy Desktop. The plugin never activates in a cloud sandbox.

Element addressing: prefer tokens, avoid coordinates

This is the key to using computer well and the easiest place to get stuck. cua-driver 0.17 has strict addressing rules:

Form	Usable	Requirements
`element_token`	Preferred	Must be accompanied by `pid`
`element_index` + `snapshot_id`	Usable	Both must come from the same `get_window_state` call, plus `pid` + `window_id`
Bare `element_index`	Refused	Returns `snapshot_id_required`
`x, y` coordinates	Usable, last resort	Depends on resolution and window position

Tokens go stale. Taking another get_window_state snapshot of the same window invalidates the previous batch, producing stale_element_token. The right response is to re-snapshot and retry, not to fall back to coordinates.

Standard workflow

list_apps or get_accessibility_tree     → find the target app's pid
list_windows(pid=...)                   → find the window_id
get_window_state(pid=..., window_id=...) → snapshot_id + elements[] with element_token
click / type / keypress / set_value     → address with element_token + pid
get_window_state again                  → verify the change (also refreshes tokens)

Step 5 is not optional. Pixel-level operations return nothing that tells you whether the click landed; only a fresh snapshot confirms it.

The 18 actions

Observation

Action	Parameters	Notes
`screenshot`	none = full screen; `pid`+`window_id` = one window	Returns an image file path
`get_window_state`	`pid`, `window_id` (both required)	AX tree + screenshot; optional `include_screenshot` (default true), `max_elements`, `query`
`get_accessibility_tree`	—	Lightweight desktop overview: running apps and visible windows
`list_windows`	optional `pid`	List windows
`list_apps`	—	Running and installed apps with state flags
`cursor_position`	—	Current cursor position
`check_permissions`	—	Reports Accessibility and Screen Recording permission status

Action

Action	Parameters	Notes
`click`	`element_token`+`pid` (preferred) / `element_index`+`snapshot_id`+`pid`+`window_id` / `x`,`y`	Optional `button`, `count`, `modifier`
`double_click`	Same as click	A dedicated action — do not pass `double_click:true` to click
`type`	`text`, `pid`; `element_token` to target	Types keystroke by keystroke
`set_value`	`pid`, `value` + `element_token`	Sets the value directly — faster than typing and immune to IME interference
`keypress`	`keys` (e.g. `["cmd","c"]`), `pid`, `window_id`	See “Background key delivery” below
`invoke_menu`	`path` (e.g. `["File","New Window"]`), `pid`, `window_id`	Goes through the accessibility menu — never steals focus, never falls back to pixels
`scroll`	`direction`, `amount`, `pid`, `window_id`	`element_token` targets a specific element
`drag`	`from_x`, `from_y`, `to_x`, `to_y`, `pid`, `window_id`	Note the parameter names: `from_` / `to_`
`launch_app`	`bundle_id` or `app_name`	Launches in background without stealing focus
`bring_to_front`	`pid` (required)	0.17 does not accept `bundle_id`; resolve the pid via `list_apps` first
`wait`	`duration_ms` (default 1000, max 30000)	Let the UI settle
`move`	—	Hard-disabled, see below

Phantom cursor: move is hard-disabled

Calling move returns this error directly:

Error: 'move' action is disabled — Profy uses phantom cursor mode
(real system cursor is never moved). Use element_index-based click
with pid + window_id instead.

This is by design, not a defect. Profy runs in phantom cursor mode:

Accessibility-addressed clicks dispatch events through the AX API and never touch the cursor
Pixel-coordinate clicks save and restore the cursor position in the driver layer

So you can keep using your computer while the expert operates the desktop; neither side fights for the pointer. keypress defaults to delivery_mode="background", which does not steal focus. The cost: native application menu shortcuts (Cmd+N, Cmd+F, and so on) may be silently ignored in the background — no error, nothing happens. The correct handling order:

Verify with get_window_state after the keypress and notice nothing changed
Retry once with delivery_mode="foreground" (this does steal focus)
Or switch to invoke_menu, which goes through the accessibility menu — neither stealing focus nor being ignored

For anything menu-driven, invoke_menu is the better default.

Automatic pixel fallback

When a token-addressed click is refused by the accessibility layer, the tool automatically performs one pixel retry: it re-snapshots, locates the element by the index parsed out of the failed token, and clicks the center of its frame. That fallback requires window_id to have been supplied. Without it, the original error is returned untouched. So pass window_id alongside token addressing even though the token alone would work.

Failures and what to do

Error	Meaning	Fix
`snapshot_id_required`	`element_index` given with no `snapshot_id`	Switch to `element_token`, or include the `snapshot_id` from the same snapshot
`stale_element_token`	The token belongs to a previous snapshot	Re-run `get_window_state` and retry; do not fall back to coordinates
`'move' action is disabled`	Hard-disabled by phantom cursor mode	Use a token-addressed click
Shortcut pressed but nothing happened	Background delivery ignored by a native app	Retry once with `delivery_mode="foreground"`, or use `invoke_menu`
`bring_to_front` does nothing	A `bundle_id` was passed; 0.17 only accepts `pid`	Resolve the pid via `list_apps` first
Computer Use missing from the plugin panel	No desktop connection	The plugin `requires: desktop_connected` and only appears on desktop
Every permission-sensitive action fails	System permissions not granted	Run `check_permissions` and grant Accessibility and Screen Recording in system settings

Execution path

agent-runtime → sidecar stdio → Electron main process → Desktop MCPServer
              → driver.ts translateCall → cua-driver (Rust)

screenshot is translated at the driver layer into get_desktop_state (full screen) or get_window_state (window-scoped) — cua-driver has no standalone screenshot command. The underlying driver is trycua/cua-driver-rs, pinned to 0.17 in the desktop platform.ts. Every addressing constraint above is the behavior of that version.

How it divides work with chrome and browser

Dimension	`computer`	`chrome`	`browser`
Target	The whole desktop	Your own Chrome, with your sessions	An isolated sandbox browser, no sessions
Perception	Screenshot + AX tree	DOM snapshot + refs	ARIA snapshot
Addressing	element_token / coordinates	CSS selector / ref	ARIA selector
Prerequisite	Desktop	Desktop, or Chrome with the extension	None — a cloud sandbox suffices
Best for	Native apps, games, cross-app flows	Sites that need you logged in	Public web research

Prefer chrome or browser for anything on the web: they do not depend on screen resolution and are far more stable. Reserve computer for cases where there is no other way in.

Keep reading

Chrome

Control your own browser, sessions included

Browser

The isolated sandbox browser

Record and replay

Turn a desktop sequence into a reusable flow

Tool catalog

Full parameters for every action

Get Started

Core Concepts

Guides

Platform & Account

Deep Dive

Computer Use

Activation

Element addressing: prefer tokens, avoid coordinates

Standard workflow

The 18 actions

Observation

Action

Phantom cursor: move is hard-disabled

Background key delivery and its blind spot

Automatic pixel fallback

Failures and what to do

Execution path

How it divides work with chrome and browser

Keep reading

Chrome

Browser

Record and replay

Tool catalog

​Activation

​Element addressing: prefer tokens, avoid coordinates

​Standard workflow

​The 18 actions

​Observation

​Action

​Phantom cursor: move is hard-disabled

​Background key delivery and its blind spot

​Automatic pixel fallback

​Failures and what to do

​Execution path

​How it divides work with chrome and browser

​Keep reading

Chrome

Browser

Record and replay

Tool catalog

Activation

Element addressing: prefer tokens, avoid coordinates

Standard workflow

The 18 actions

Observation

Action

Phantom cursor: move is hard-disabled

Background key delivery and its blind spot

Automatic pixel fallback

Failures and what to do

Execution path

How it divides work with chrome and browser

Keep reading