Expert Evolution

One-line definition

A published expert is not static. In the background the system reads real conversation trajectories and measured execution cost, rewrites the expert’s skill documents, and verifies through a 10-dimension score gate that “the next run is cheaper and no worse”. Only proposals that clear the gate appear in your evolution panel. You decide whether to accept. The system never changes your expert on its own.

What this is, and what it isn’t

“AI self-improvement” now covers a very wide range, so it is worth drawing the boundary first.

	Recursive self-improvement (RSI)	Profy expert evolution
What improves	The model-building process itself	The expert’s skill documents (procedural memory)
How it takes effect	Automatic, recursive	Single-pass, non-recursive, human acceptance required
Acceptance criteria	Usually no explicit gate	Dual-ratchet hard gate, enforced in code
Underlying model	Changes	Unchanged

What Profy does is targeted optimization of procedural memory. An expert accumulates knowledge across real tasks about which paths work and which are dead ends; evolution writes that back into the skill document so the next run does not have to rediscover it. Model weights, reasoning capability, and platform code are all outside the scope. The distinction matters because it determines the risk surface: what changes is a human-readable, diffable, revertible Markdown document — not an uninterpretable weight update. What you see in the panel is the complete change.

On the currently much-discussed question of how to design an evaluation gate that cannot be reward-hacked, Profy’s answer is the dual ratchet plus one code-owned dimension described below. Its value is not sophistication — it is that it closes the two easiest routes around a gate. See “why it is built this way”.

The objective function: cheaper and no worse

The objective is the first paragraph of the system prompt, and it is one sentence:

Make the NEXT run cheaper and not worse. Experience must REPLACE exploration, not accumulate on top of it.

That sentence rules out an entire class of changes that look like improvements. Adding defensive steps, fallbacks, or richer prose is not progress unless it removes more cost than it adds. This stance has a cost and it is deliberate. A steadily thickening skill document intuitively reads as “the expert learned more”, but it is injected into context in full on every run — a fixed cost that grows with the document. Thicker usually means more expensive.

The rubric: 10 dimensions, two groups

Each dimension is scored 0–100. The seven quality dimensions carry a combined weight of 0.70 and the three efficiency dimensions 0.30, keeping the total on a 0–100 scale.

Quality group (0.70)

Dimension	Weight	Score 0	Score 50	Score 100
`workflow_clarity`	0.14	Prose without steps	Numbered steps, ambiguous ordering/branching	Unambiguous sequence, explicit inputs/outputs per step
`executable_specificity`	0.14	Abstract advice (“search properly”)	Concrete but incomplete commands	Copy-pasteable commands, exact endpoints and field names
`empirical_grounding`	0.12	Speculative content	Partially grounded in the trajectory	Every claim traceable to observed runs
`failure_encoding`	0.10	No known failures recorded	Failures mentioned without recovery	Each failure has a concrete symptom and the exact recovery action
`architecture_fit`	0.07	Contradicts platform constraints	Works but ignores available primitives	Uses platform tools and skills idiomatically
`antipattern_coverage`	0.07	No “do NOT” guidance	Generic warnings	Specific antipatterns observed in real runs, each with the correct alternative
`frontmatter_quality`	0.06	Missing or wrong name/description	Valid but vague description	Precise trigger-oriented description the router can act on

Efficiency group (0.30)

Dimension	Weight	Criterion
`step_economy`	0.12	Does the SOP reach the result in the fewest steps? 0: the SOP adds steps (mandatory double-checks, redundant validation, “always verify X twice”); 50: neutral; 100: the SOP eliminates steps the agent previously needed (direct commands replace search→read→try loops). Adding fallback or re-verification steps lowers this score.
`exploration_replacement`	0.10	Does the SOP replace exploration with known answers? 0: the agent must still discover endpoints and formats; 100: known endpoints, response shapes, and gotchas are stated directly with explicit “do not re-verify” guidance where results are stable. Encoding “try A, then B, then C” instead of the known-good answer lowers this score.
`context_footprint`	0.08	Scored automatically by the system — see below

context_footprint: the one dimension the model does not score

This dimension’s score is computed in code from the document’s character count. Any value the model passes is discarded:

Document length	Score
≤ 3,000 characters	100
3,000 – 24,000 characters	Linear decay
≥ 24,000 characters	0

The formula is 100 × (24000 − chars) / 21000, clamped to 0–100. The thresholds are measured, not invented. The Stage 4 baseline had a 429-character seed skill that evolved into 12,109 characters of output. So 3,000 characters is defined as a compact injectable SOP (full marks), and 24,000 — roughly twice the Stage 4 output, approaching a runaway fixed per-run cost — scores zero. When the current skill content cannot be fetched (a failed lookup, say), the “before” score falls back to a neutral 50.0. That neither rewards nor punishes, and keeps the ratchet decidable instead of erroring out.

Why this dimension must be computed in code: the skill document is injected into context in full on every run, so its size is a fixed per-run input cost. If the model scored its own document length, it would almost always conclude “yes it got longer, but it is more thorough now”. This is exactly where self-assessment structurally fails, so this dimension was taken out of self-assessment.

The dual ratchet

A suggestion is accepted only if both conditions hold:

Quality group must not decrease

weighted quality score (after) ≥ weighted quality score (before)

Efficiency group must strictly increase

weighted efficiency score (after) > weighted efficiency score (before)

Compensation is forbidden in both directions

This is the core of the design and what distinguishes it from a simple “weighted total must go up” rule. On a quality regression the tool returns:

Ratchet rule violation (quality regression): quality group score dropped
<before> → <after>. Efficiency gains cannot compensate quality loss —
the improvement must keep quality at least equal.

On flat or worse efficiency:

Ratchet rule violation (efficiency must strictly improve): efficiency group
score went <before> → <after>. Quality gains cannot compensate — the next run
must be cheaper (fewer steps, less exploration, smaller context footprint).

Both messages carry the same clause: neither may compensate the other.

Why it is built this way

A single weighted-total gate has a structural hole: any dimension can be paid for by another. If all you require is that the total rises, a change scoring +15 on quality and −10 on efficiency passes easily — and that is precisely the most common degeneration. The document grows thicker and more “thorough”, every run gets more expensive, but every quality metric climbs and the total looks healthy all the way down. Splitting into two groups and requiring “one must not fall, the other must strictly rise” closes that route. It also closes the reverse: you cannot buy efficiency by deleting quality content, because the quality group is not allowed to drop. The second hole is self-assessment. If all ten dimensions were scored by the model, the gate’s evidence and the thing being gated would share one source — it passes whenever it wants to. context_footprint, computed in code from character count, is the one anchor on that chain the model cannot fabricate. Its weight is only 0.08, but because the efficiency group must strictly increase and contains just three dimensions, a large document expansion usually drags the whole group down enough to block the change.

The two holes are two faces of one problem: the evidence a gate relies on cannot be supplied by the thing it guards. In the first, dimensions supply each other’s excuses. In the second, the model writes its own report card.

The legacy 7-dimension rubric

Setting EVOLUTION_RUBRIC=quality switches back to the old seven dimensions (quality group only, weights renormalized to 1.0) and degrades the rule to “weighted total must strictly increase”. This exists for rollback only. The default everywhere is quality_eff.

Measured cost as ground truth

Evolution does not only read conversation trajectories. It also receives a <run_stats> block listing the measured cost of past runs:

Field	Meaning	Bound
`run_index`	Which run	0–1000
`total_tokens` / `input_tokens` / `output_tokens`	Token consumption	≤ 10,000,000 each
`llm_turns`	Model turns	≤ 200
`tool_calls`	Tool call count	≤ 500
`skill_chars`	Skill document length at the time	≤ 200,000
`success`	Whether it succeeded	boolean
`skill_version_note`	Version note	≤ 120 chars

At most 20 entries. The interpretation rules are fixed in the skill document:

Tokens went up after the skill grew → the previous direction was wrong; this round must compress and delete rather than enrich
Tokens went down and success held → the direction was right; keep encoding shortcuts while watching document size
Never argue against run_stats with theory; measured cost wins

This separates “I think this is better” from “this is measurably cheaper”. Without it, the rubric would be a closed loop of self-persuasion.

run_stats is a strict whitelist model with per-field bounds because it is injected verbatim into the evolution prompt. Unvalidated input would both bloat context and give callers a way to smuggle in instructions.

Triggering and scheduling

When a run happens

Evolution does not run every turn. The counter accumulates per (user, expert) pair across sessions:

Parameter	Default	Notes
`MEMORY_EVOLUTION_INTERVAL`	5	Interactions before a run triggers
Counter storage	Redis	Key `evo_counter:{user_id}:{expert_identifier}`, TTL 7 days
When Redis is down	In-process dict fallback	Resets on restart; acceptable

Counting per user-and-expert rather than per session is deliberate: one user may open many short sessions of one or two messages each with the same expert, and a per-session counter would never reach the threshold. The 7-day TTL means a week of silence resets it — acceptable for a nudge-style trigger.

Platform-side scheduling gates

Beyond conversation triggers there is a periodic scheduler (disabled by default):

Environment variable	Default	Purpose
`EVOLUTION_SCHEDULER_ENABLED`	`false`	Master switch
`EVOLUTION_SCHEDULER_INTERVAL_MS`	3,600,000 (1 hour)	Scan interval
`EVOLUTION_COOLDOWN_HOURS`	24	Minimum gap between runs for one expert
`EVOLUTION_FAILURE_BACKOFF_HOURS`	6	Backoff after a failure
`EVOLUTION_TRIGGER_THRESHOLD`	3	Trigger threshold
`EVOLUTION_PENDING_MAX`	5	Stop generating once this many suggestions are pending
`EVOLUTION_BATCH_SIZE`	10	Experts per batch
`EVOLUTION_CONCURRENCY`	1	Concurrency
`EVOLUTION_TIMEOUT_SEC`	900	Per-run timeout (Stage 4 measured 721–1003s)
`EVOLUTION_RECENT_SESSION_LIMIT`	5	Recent sessions used as material
`EVOLUTION_RECENT_MESSAGE_LIMIT`	40	Recent messages used

EVOLUTION_PENDING_MAX = 5 is the one you will feel directly: once five suggestions are unhandled, no new ones are generated for that expert. Clearing the panel regularly is not tidiness, it is the precondition for evolution continuing at all.

Capability boundary of evolution mode

Evolution runs in an isolated internal invoke, not as part of your normal conversation.

Constraint	Value
Activation	`requires: { is_internal_invoke: true }`
Exclusivity	`exclusive: true` — no other plugin loads in evolution mode
Available tools	Only `skill` (read-only), `memory`, `evolution_log`
Blocked tools	`skill_manage`, sandbox tools, image, video, delegate, bash, and every write/execute tool
User-facing output	None — evolution mode produces no user-facing messages
Submissions per invoke	At most 1
Improvable targets	`expert/*` scoped skills only

Scope is validated, not merely conventional

Before submission the tool confirms the target really is expert/*. If not, it refuses:

Error: '<name>' is not an expert-scoped skill. Evolution can only improve
expert/* skills. Available expert skills: <list>

builtin/* (platform built-ins) and user/* (the user’s global library) are both out of scope. Changing the former would be changing the platform; the latter does not belong to this expert. The skill_name must also be bare (docx, not expert/docx); a scope prefix is rejected.

Optional memory consolidation

Evolution mode may also call memory(action="dream", scope="expert") to consolidate memories (merge duplicates, archive stale entries). It is optional and low priority — only when skill improvement has already been submitted or skipped and the token budget has room. Skill evaluation always takes precedence.

Accepting and ignoring

What acceptance does

Validate the suggestion structure

An invalid structure returns 400 and nothing is written.

Locate the target skill

Looked up by expertIdentifier, shared scope, and skill name. A miss returns 404 — meaning the skill was deleted after the suggestion was generated.

Three things in one transaction

Update the skill content, mark the suggestion accepted, and increment the expert’s evolutionCount. All in one transaction, so “skill changed but not counted” cannot happen.

Accepted changes take effect immediately for all subsequent conversations with no re-review — a skill improvement is an internal optimization that does not change the capability description you published. Only the expert’s creator can accept. Anyone else gets 403 (Only the expert creator can accept suggestions).

What ignoring does

It changes the suggestion status and never touches skill content. Ignored suggestions do not reappear. When you are unsure, ignoring is the safe default — if the direction genuinely has value it will return in another form from newer conversation data.

What the panel shows

Field	Content
Target skill	The `expert/*` skill being improved
Summary	One-line description of the change
Diff summary	What changed and why
Improved content	The complete new SKILL.md (not a diff fragment)
Score comparison	Before/after across 10 dimensions plus quality, efficiency, and total
Improved dimensions	Which dimensions rose

On success the tool returns:

Suggestion submitted (id: <id>). Score: <before> → <after> (+<delta>).
Creator will review before any skill changes take effect.

How to read a suggestion

Read the efficiency group first

The efficiency group is the one required to strictly rise, so its delta tells you how much this change actually saves. The quality group only guarantees no regression. A suggestion whose efficiency group moved 0.5 points is probably marginal.

Check whether context_footprint went up or down

This is the one dimension that cannot be self-scored. A rise means the document was compressed — usually a good sign, since compressing is much harder than expanding. A fall alongside a rising efficiency group means step_economy or exploration_replacement gained enough to offset it; those changes are worth reading closely to see exactly what was removed and what was added.

Read the full improved content, not just the summary

The panel gives you the complete new document. A skill document is injected in full, so one wrong sentence will be followed in every future conversation. Ten minutes of reading costs far less than diagnosing it afterwards.

Product positioning is yours to guard

The rubric measures whether the document is well written and cheap to run. It knows nothing about your product positioning. A suggestion that is technically sound but pulls the expert’s style away from what you want should be ignored without hesitation — that is structurally invisible to the rubric.

Do not let the panel pile up

At five pending suggestions, generation stops. A backlog means evolution has halted.

Boundaries and failure modes

Symptom	Cause	Fix
No new suggestions for a long time	Pending count hit `EVOLUTION_PENDING_MAX` (default 5)	Clear the panel
Low-usage expert never gets suggestions	The (user, expert) counter never reaches `MEMORY_EVOLUTION_INTERVAL` (default 5), and the 7-day TTL resets it	Expected; they arrive as usage grows
404 on accept	The skill was deleted after the suggestion was generated	Ignore the suggestion
403 on accept	You are not the expert’s creator	Only the creator can accept
The score barely moved	The ratchet only requires a strict increase, with no minimum	Marginal suggestions can simply be ignored
Suggestions never touch a built-in skill	Evolution only modifies `expert/*`	By design; built-in changes require a platform release
Expert behavior unchanged after accepting	The document changed but conversations never activate that skill	Check whether the frontmatter description lets the router match it

Evolution changes skill documents — not your persona / soul / agent configuration, and not the model. To change your expert’s personality or voice, use the prompt layer configuration in Studio; that is outside evolution’s scope.

Evolution vs. memory

Two mechanisms that are often conflated but act on entirely different things:

Dimension	Evolution	Memory
Acts on	The expert’s skill documents (how to do things)	Facts from interactions (what happened)
Approval	Manual acceptance required	Accumulates automatically
Reach	All future conversations	Contextually relevant conversations
Gate	Dual-ratchet scoring	Quality gating plus consolidation
In essence	The expert’s method got better	The expert remembered something

Memory lets an expert know specific people and situations; evolution makes its general method cheaper and more accurate. They run independently and do not overwrite one another.

How evolution works

The evolution pipeline from a system perspective

Expert memory

The memory mechanism, orthogonal to evolution

Analytics

Measuring the effect of evolution with data

Expert configuration

The part evolution does not touch: prompt layering

Verified 2026-08-11. Sources: services/agent-runtime/src/plugins/builtin/evolution/tools/evolution_log.py, skills/evolution/SKILL.md, prompts/EVOLUTION.md, hooks/background_review.py, services/agent-runtime/src/models/evolution.py, services/core/src/db/service/expert-evolution.ts, services/core/src/config/env.ts. Decisions: knowledge/decisions/adr-0024-darwin-efficiency-objective.md, adr-0015-darwin-evolution-p0-boundary.md.

Overview

Create & Configure

Expert Configuration

Publish & Operate

Revenue & Analytics

Tools

Expert Evolution

Expert Evolution

One-line definition

What this is, and what it isn’t

The objective function: cheaper and no worse

The rubric: 10 dimensions, two groups

Quality group (0.70)

Efficiency group (0.30)

context_footprint: the one dimension the model does not score

The dual ratchet

Compensation is forbidden in both directions

Why it is built this way

The legacy 7-dimension rubric

Measured cost as ground truth

Triggering and scheduling

When a run happens

Platform-side scheduling gates

Capability boundary of evolution mode

Scope is validated, not merely conventional

Optional memory consolidation

Accepting and ignoring

What acceptance does

What ignoring does

What the panel shows

How to read a suggestion

Boundaries and failure modes

Evolution vs. memory

How evolution works

Expert memory

Analytics

Expert configuration

​Expert Evolution

​One-line definition

​What this is, and what it isn’t

​The objective function: cheaper and no worse

​The rubric: 10 dimensions, two groups

​Quality group (0.70)

​Efficiency group (0.30)

​context_footprint: the one dimension the model does not score

​The dual ratchet

​Compensation is forbidden in both directions

​Why it is built this way

​The legacy 7-dimension rubric

​Measured cost as ground truth

​Triggering and scheduling

​When a run happens

​Platform-side scheduling gates

​Capability boundary of evolution mode

​Scope is validated, not merely conventional

​Optional memory consolidation

​Accepting and ignoring

​What acceptance does

​What ignoring does

​What the panel shows

​How to read a suggestion

​Boundaries and failure modes

​Evolution vs. memory

​Related pages

How evolution works

Expert memory

Analytics

Expert configuration

Expert Evolution

One-line definition

What this is, and what it isn’t

The objective function: cheaper and no worse

The rubric: 10 dimensions, two groups

Quality group (0.70)

Efficiency group (0.30)

context_footprint: the one dimension the model does not score

The dual ratchet

Compensation is forbidden in both directions

Why it is built this way

The legacy 7-dimension rubric

Measured cost as ground truth

Triggering and scheduling

When a run happens

Platform-side scheduling gates

Capability boundary of evolution mode

Scope is validated, not merely conventional

Optional memory consolidation

Accepting and ignoring

What acceptance does

What ignoring does

What the panel shows

How to read a suggestion

Boundaries and failure modes

Evolution vs. memory

Related pages