Skip to main content
Desktop Automation is a capability exclusive to Profy Desktop — Expert can operate desktop applications just like you do: moving the mouse, clicking buttons, typing text, and reading screen content to help you automate repetitive work on the desktop.
Desktop Automation is only available in the Profy Desktop client and supports macOS, Windows, and Linux. This capability is not available in the web version.

​
Platform Support

The underlying driver is powered by trycua/cua-driver-rs, communicating through a unified MCP protocol interface across all three platforms with consistent tool behavior.

​
Core Capabilities

Mouse Control

Move the cursor, single-click, double-click, right-click, drag — Expert can precisely operate any visible element on the desktop.

Keyboard Input

Type text in any application, execute keyboard shortcuts, and switch input methods.

Screen Understanding

Expert can “see” your screen content, understand the interface layout and current state, and make operation decisions accordingly.

Cross-Application Coordination

Switch between multiple desktop applications — copy files from Finder to an email client, or extract data from a browser and paste it into a spreadsheet application.

​
Phantom Cursor

Profy Desktop uses a phantom cursor — the Expert operates in the background without moving your real mouse.
  • Cursor position protection: Your cursor position is saved before each action and restored afterwards
  • Move action blocked: The move action is hard-blocked — Expert can only click to jump directly to targets
  • Collision Watch: If you move your mouse during Expert’s operation, the system auto-pauses to avoid conflicts
  • Emergency Stop: Press ESC at any time to immediately halt all desktop automation
This aligns with Claude Desktop’s phantom cursor behavior — AI operations and user operations share the same computer without interfering with each other.

​
How to Use

1

Open Profy Desktop

Make sure you’re using the Profy Desktop client (not the web version).
2

Describe the Task

Tell Expert in the conversation what you’d like to accomplish on the desktop. For example: “Open System Preferences, find the Display options, and change the resolution to default.”
3

Observe and Confirm

Expert will begin operating your desktop. You can watch the mouse movement and operation process in real time. If a step requires your confirmation, Expert will pause and ask.
4

Get Results

Once the task is complete, Expert will report the results in the conversation.

​
Typical Use Cases

Rename a batch of files, organize folders by rules, or batch-modify file properties — tell Expert the rules and it will execute them one by one.
Have Expert complete complex configuration changes in system settings or application preferences, saving you the time of navigating menus.
Extract data from one application, convert the format, and input it into another — for example, copying table data from a PDF reader to Excel.
Any mouse-and-keyboard operation sequence that you need to perform repeatedly can be delegated to Expert.
Unfamiliar with how a desktop application works? Have Expert demonstrate while explaining each step in the conversation.

​
Security

Desktop Automation has the following protections:
  • Sandbox Isolation (macOS): Expert’s operations are restricted by a system-level sandbox (Seatbelt), preventing access to sensitive system areas
  • Visible Operations: All operations are visible on your screen in real time, and you can interrupt at any point
  • Limited Permissions: Expert cannot access Keychain, password managers, or other security-sensitive areas
  • Collision Watch: The system detects user mouse movement during Expert operations and automatically pauses
  • Emergency Stop (ESC): Press ESC at any time to immediately stop all desktop automation
  • Per-App Approval: Sensitive apps (System Preferences, Terminal, Keychain) are blocked; unknown apps require confirmation before first use
Windows and Linux do not have system-level sandbox isolation (Seatbelt is macOS-specific) — commands are executed directly. Windows uses PowerShell, Linux uses bash.

​
Image Pipeline

To reduce model token cost and speed up understanding, screenshots go through an optimization pipeline:
  • JPEG Compression: Screenshots are automatically converted from PNG to JPEG (max 1280px width, 80% quality), reducing file size by 60-80%
  • Apple Vision OCR (macOS): Uses the native Vision Framework to extract text from screenshots without cloud API calls
  • Post-action Sleep: A configurable 100ms delay after each action lets the UI settle before the next screenshot
These optimizations are transparent and automatic.

​
App Approval

Desktop automation enforces per-app permission levels: You can customize per-app permissions in Settings.

​
Recommendations

  • Keep the screen clean: Expert “sees” the screen to understand its targets — too many elements on the interface can increase recognition difficulty
  • Specify the target application: Tell Expert exactly which application to operate rather than letting it search
  • Close obstructing windows: Make sure the target application’s window is not blocked by other windows
  • Guide step by step: Break complex operation flows into step-by-step instructions and confirm each step before continuing

​
Settings

In the Profy Desktop settings panel, you can configure desktop automation behavior:
  • Any App: Whether to allow operation on all desktop apps
  • Chrome Control: Whether to allow control of the user’s real Chrome browser
  • Post-action Sleep: Wait time before screenshots (adjustable 50–500ms)
  • OCR: Whether to enable text extraction from screenshots
  • Collision Watch: Whether to auto-pause when user mouse movement is detected

​
Browser Automation vs. Desktop Automation

These two capabilities complement each other — use Browser Automation for web-related tasks and Desktop Automation for local application operations.

​
Billing

Credit consumption for Desktop Automation primarily comes from model calls and screenshot analysis in the conversation. The operations themselves incur no additional charges.