Computer Use

computer(action=...) provides pixel-level screen control, suitable for operating any GUI application.

Overview

Computer Use is the lowest-level interaction tool — it observes the screen via screenshots and operates through coordinates and keyboard input. Suitable for native apps, games, and any GUI interface without DOM structure.

Available Actions

Action	Description	Required Params
`screenshot`	Capture screen/window	`window_id` (optional)
`click`	Click at coordinates	`x`, `y` or `element_index`
`double_click`	Double-click	`x`, `y`
`type`	Type text at cursor	`text`
`scroll`	Scroll	`direction`, `amount`
`drag`	Drag from start to end	`start_x`, `start_y`, `end_x`, `end_y`
`keypress`	Key combination	`keys` (array)
`move`	Move cursor	`x`, `y`
`cursor_position`	Get cursor position	—
`list_windows`	List all windows	—
`get_window_state`	Get accessibility tree	`window_id` or `pid`

Examples

Screenshot

computer(action="screenshot")

Click at coordinates

computer(action="click", x=500, y=300)

Keyboard shortcut

computer(action="keypress", keys=["cmd", "c"])  # Copy
computer(action="keypress", keys=["cmd", "v"])  # Paste

Window management

computer(action="list_windows")
computer(action="get_window_state", window_id=12345)
computer(action="click", element_index=3, window_id=12345)

Execution Backends

Environment	Backend	Notes
Cloud Sandbox	noVNC pixel stream	Screenshots and control via VNC
Desktop	macOS CUA daemon	Native Accessibility API

Best Practices

Screenshot before action: Always screenshot to confirm UI state
Prefer element_index over coordinates: More reliable than coordinates
Wait for response: Wait for UI updates after complex operations
Window focus: Ensure target window is in foreground

Activation

computer is only available in Desktop mode (desktop_connected=true).

Rendering Surfaces Browser Automation

Get Started

Core Concepts

Chat with Experts

Creation & Automation

Plugin Tools

Memory & Personalization

Marketplace & Content

IM Channels

Account & Billing

Team

How It Works

Help

Computer Use

Computer Use

Overview

Available Actions

Examples

Screenshot

Click at coordinates

Keyboard shortcut

Window management

Execution Backends

Best Practices

Activation

​Computer Use

​Overview

​Available Actions

​Examples

​Screenshot

​Click at coordinates

​Keyboard shortcut

​Window management

​Execution Backends

​Best Practices

​Activation

Computer Use

Overview

Available Actions

Examples

Screenshot

Click at coordinates

Keyboard shortcut

Window management

Execution Backends

Best Practices

Activation