Record and Replay

Record and Replay lets you demonstrate a real workflow to an Expert: you do it the way you normally would, the Expert captures events, screenshots and your spoken narration in the background, and then turns that demonstration into a reusable capability.

This is a desktop-only capability: the plugin activates only when desktop_connected: true, so it is neither visible nor usable on the web. Native event-stream recording is macOS only; Windows and Linux use CUA trajectory recording.

Two paths with entirely different semantics

This is the easiest thing to conflate — “recording” produces one of two very different artifacts:

Path	Artifact	How it replays	Platform
Native event stream → skill	A reusable skill	Semantic replay: elements are re-resolved at run time, so layout changes are survivable	macOS only
CUA trajectory recording	A set of `turn-NNNNN/` folders	Mechanical replay: the recorded tools are re-invoked with the recorded arguments	macOS / Windows / Linux

The rule of thumb: pick a skill for long-term reuse, pick a trajectory for exact reproduction. Skills adapt to change but don’t guarantee step-identical behaviour; trajectories are step-identical but break the moment the UI moves.

Path 1: record → skill (macOS)

What actually happens

Start recording

You say “record what I’m about to do” and the Expert calls event_stream_start. A floating “Recording” pill with a Stop button appears on screen.Recording is a background system process: it captures clicks and typing across all apps and all windows. Switching screens, switching apps and minimising the chat window do not interrupt it.

Narrate while you work

If the tool returns audioRecording: true, the microphone is capturing your narration too. This is worth doing — stated intent is more reliable than intent inferred from clicks, and the resulting skill is noticeably better.

Stop

Click Stop on the floating pill (preferred), or come back and say you’re done. Both routes lead into analysis.

The Expert analyses and creates a skill

The Expert reads the event file, the screenshots and the audio transcript, works out what you did and why, then persists a skill via skill_manage(action="create"). Recording isn’t finished until that call succeeds.

Use it next time

In a new conversation just name the task; the Expert follows the skill and re-resolves buttons and fields at run time rather than replaying the original coordinates.

What a recording produces

The session directory lives at ~/.profy/recordings/{sessionId}/:

~/.profy/recordings/<uuid>/
├── events.jsonl        # one event per line: app_switch / mouse_click / key_down / scroll …
├── session.json        # session metadata: start, end, end reason, event count
└── snapshots/
    ├── <timestamp>-<trigger>.png    # screenshot at a key moment
    └── <timestamp>-<trigger>.json   # the accessibility (AX) tree at that same moment

These are ordinary files on disk, read by the Expert with the read tool. events.jsonl routinely runs to thousands of lines, and reading it whole would blow the context window — so the Expert samples it in 100-line slices (start, end, then filling gaps), focusing on app_switch, mouse_click and key_down while skipping scroll and mouse_move noise.

Path 2: CUA trajectory recording and replay (cross-platform)

Trajectory recording never turns itself on — you have to ask for it explicitly. Once enabled, every action tool call (click, right_click, scroll, type_text, press_key, hotkey, set_value) writes a numbered folder; read-only tools (screenshot, list windows, get window state) are not recorded.

cua-driver recording start ~/cua-trajectories/run-1
cua-driver recording status
cua-driver recording stop

Each action writes into turn-NNNNN/ (five-digit zero-padded counter):

File	Contents
`action.json`	Tool name, full input arguments, result summary, pid, click point, ISO-8601 timestamp
`app_state.json`	Post-action accessibility snapshot
`screenshot.png`	Post-action window capture
`click.png`	Click-family actions only: the screenshot with a red dot at the click point

Replay uses replay_trajectory({dir}): it walks turn-NNNNN/ in lexical order, reads each action.json, and re-invokes the same tool with the recorded arguments. Two knobs: delay_ms (pacing between turns, default 500) and stop_on_error (halt on first failure, default true).

Element addressing does not survive across sessions. cua-driver 0.17 refuses stale element_token values and rejects bare element_index outright. Which means:

Pixel clicks (click({pid, window_id, x, y})) and keyboard tools replay cleanly
Element-addressed actions need a fresh get_window_state, which mechanical replay never re-emits

For reliable replay, compose trajectories from pixel and keyboard primitives; for adaptability, use the skill path, where semantic replay re-resolves elements at run time.

When to use it

Turn repetitive work into one sentence

Export data weekly, reformat it, send the report? Record once, then trigger it with a sentence. This is the canonical use of the skill path.

Teach the Expert an app it doesn't know

Internal systems and niche tools have no public documentation; demonstrating once beats explaining ten times. Narration pays off most here.

Regression checking

Record a trajectory as a baseline, then re-run after a change and compare state snapshots. This is the trajectory path’s home turf — regression diffs are what it was designed for.

Training data collection

Every turn folder is naturally a (state, action, next_state) triple.

Built-in app skills

The plugin ships operating skills for six common apps: Clock, Numbers, Spotify, Notion, Music and iPhone Mirroring. You don’t need to record those yourself.

Key numbers

Item	Value
Maximum recording length	30 minutes (auto-stops, end reason recorded as `timeout`)
Concurrent recordings	1 (a new one cannot start while another runs)
Trajectory replay pacing	500 milliseconds between turns (default, adjustable)
Trajectory replay on error	Halt immediately (default, adjustable)
Event-file read granularity	100 lines per read (agent-side convention, prevents context overflow)
Skills shipped with the plugin	9 (2 record/replay + 1 CUA driver + 6 app skills)

Failures and fixes

Symptom	Cause	Fix
No “Record and Replay” in the plugin panel	The plugin requires a desktop connection (`desktop_connected: true`)	Use the Profy desktop app; the web app doesn’t offer it
”recorder tools require the Profy Desktop app”	The sidecar isn’t connected	Confirm the desktop app is running and signed in
It says macOS only	Native event streaming (profy-event-tap) is macOS-specific	Use CUA trajectory recording on Windows / Linux
Recording starts but `events.jsonl` is empty, or “Capture process exited”	Accessibility permission not granted — the capture process can’t attach	System Settings → Privacy & Security → Accessibility, enable Profy, start again
”A recording is already in progress”	Only one recording at a time	Use the active one, or stop it first
Recording stopped by itself with end reason `timeout`	It hit the 30-minute cap	Split long workflows into several recordings
The Expert says the recording isn’t enough to build a skill	Intent isn’t visible in the events (all scrolling and mouse movement)	Re-record with narration on, saying why as you go
Replay breaks at a step with a stale-element error	Element addressing doesn’t survive across sessions	Switch to pixel and keyboard primitives, or use the skill path
Replay runs too fast for the UI to keep up	The 500 ms default pacing is too tight	Have the Expert raise `delay_ms`
Password steps can’t be replayed	Recording does not retain passwords in plaintext	Type it manually before replay, or let a password manager fill it

Replay performs real actions on your real desktop. Confirm you have permission on any third-party system involved, and do a dry run before a real one — it will not judge for you whether an action is reversible.

Billing

Recording itself costs no credits (capture happens locally). What costs credits is the Expert’s model usage while analysing the recording and while replaying — reading event files and looking at screenshots are tokens, and a long recording costs noticeably more to analyse than a short one, which is another reason to record in segments.

Verified 2026-08-12. Sources: services/agent-runtime/src/plugins/builtin/record-and-replay/ (plugin.json, tools/event_stream.py, skills/record-replay.md, skills/cua-recording.md) and apps/desktop/src/main/lib/recorder/event-stream.ts.

Computer Use

How Experts operate desktop applications

Skills

How skills are stored and invoked

Get Started

Core Concepts

Guides

Platform & Account

Deep Dive

Record and Replay

Two paths with entirely different semantics

Path 1: record → skill (macOS)

What actually happens

What a recording produces

Path 2: CUA trajectory recording and replay (cross-platform)

When to use it

Key numbers

Failures and fixes

Billing

Computer Use

Skills

​Two paths with entirely different semantics

​Path 1: record → skill (macOS)

​What actually happens

​What a recording produces

​Path 2: CUA trajectory recording and replay (cross-platform)

​When to use it

​Key numbers

​Failures and fixes

​Billing

​Related

Computer Use

Skills

Two paths with entirely different semantics

Path 1: record → skill (macOS)

What actually happens

What a recording produces

Path 2: CUA trajectory recording and replay (cross-platform)

When to use it

Key numbers

Failures and fixes

Billing

Related