Choose a learning method#
Choose the editable adapter content first, then a method that understands it. A template defines the files, exports, and checks; an Evolver defines how evidence becomes an update. Sharing the interface does not make all template–method combinations meaningful.
Goal |
Starting method |
Template |
Additional inputs |
|---|---|---|---|
Improve a short instruction document with few moving parts |
|
Codex CLI, Linux learner environment |
|
Let a coding agent edit several files, skills, or code |
A template with the intended learnable paths |
Authenticated learner CLI |
|
Search text candidates with validation-based selection |
Start with prompt-only |
Validation split and the |
|
Make bounded skill edits with a validation gate |
Start with prompt-only or the paper seed |
Validation split and the |
|
Refine a harness per task from trajectories without labels, with a library of adopted harnesses |
|
Learner model endpoint, Codex, the |
|
Refine a policy, sub-agents, skills and memory in place from each step’s trajectories, without a gate |
|
Learner model endpoint, Codex |
|
Grow an itemised playbook by incremental deltas from reflections, keeping the best validation playbook |
|
Learner model endpoint; optional validation split |
|
Evolve an agent’s harness (prompt, middleware, scripts, skills, sub-agents, memory) from k rollouts per task, with each change attributed and reversible |
|
An acting agent that runs hooks (Codex, Claude Code), a learner CLI (Codex, Claude Code or Pi), k rows per task |
|
Search a dual-surface defense (prompt + tool hooks) for a frozen victim against frozen attack cases |
|
Hook-bridge pipeline, utility/attack judge (verdicts in |
|
Search instructions with a model that reads the best earlier instructions with their training scores |
|
Learner model endpoint; validation split for |
|
Build a persistent world model from an ordered environment stream |
|
Evidence store, deployment description, tools, Linux learner environment, a learner CLI (Codex, Claude Code or Pi) |
|
Evolve a system prompt, rule bank, safety memory and tool policy from diagnosed trajectories |
|
Pinned SHE library, Docker hook bridge, a training split measured whole each round ( |
|
Learn a bank of reflections online, retrieved per instance of an ordered stream |
|
The run’s |
|
Search whole adapters with a coding-agent proposer that reads every earlier candidate’s files, scores and trajectories |
|
A training split (the search set), a proposer CLI (Codex, Claude Code or Pi), an acting agent that delivers the adapter |
|
Evolve whole adapters round by round under regularizers against overfitting the evolve set: an annealed edit budget, the edit history, exploration when stalled, a pre-evaluation leakage critic, a noise-adjusted floor and a cost rule |
|
A training split (the evolve set, k rows per task), a proposer CLI (Codex, Claude Code or Pi), the |
|
Learn the framework contract offline |
Prompt-only with a teaching learner |
Included toy tasks |
|
Grow a library of verified skill entries from robot tasks, each proposal re-solved by a fresh actor before it is admitted, with the coordinator choosing the next task |
|
A coordinator CLI (Codex, Claude Code or Pi), verifier task rows in |
A learner model endpoint above can be your Claude subscription, and a learner CLI can run on it, with no API key: Claude subscription says which roles go which way.
Compare the work behind each step#
All methods share the main acting batch. The table describes additional work, excluding retries and any task-specific service overhead. Call counts vary with evidence, candidates, and settings.
Method |
Additional model / acting work |
Persistent learner state |
Final selection |
|---|---|---|---|
Prompt-only |
One Codex learner session per step that rewrites |
Notes the session keeps in |
Current adapter |
Codex / Claude |
A multi-action learner session; candidate tests depend on what the learner does |
Saved files and optional supported session handle |
Current adapter unless the method customizes finalization |
GEPA |
Initial-parent validation, reflection, child minibatch tests, and validation for a child that passes the minibatch criterion |
Candidate pool, validation scores, per-candidate next file |
Highest retained validation mean |
SkillOpt |
Reflection/merge/ranking calls, validation (cached by skill content), optional epoch-end and final validation work |
Current/best skills, validation cache, update history and meta skill |
Retained best, with optional final promotion |
Harness² |
Two editors, a composer and a reflector per task, plus three probe executions (k = 1); the adapter stays H0 |
Library of adopted harnesses with lessons |
Unchanged adapter; the stream’s own deliverables are the result |
Continual Harness |
Four Refiner calls per step (policy, sub-agents, skills, memory); no candidate rollout, no validation |
Step log in |
Current adapter |
ACE |
Per sample: one Reflector call, or for a wrong answer up to |
|
Best validation playbook, else the current one |
AHE Adapter |
Per step: one Agent Debugger session per task (up to three attempts) and one Evolve Agent session; no candidate rollout, no validation |
|
Best-ever iteration by training pass rate, or the last |
EvoSafeHarness |
Per trial: Designer, Criticizer and Analyzer calls, plus candidate runs on the training cases (and the parent’s validation cases when a candidate qualifies); the parent reuses the step’s rollouts when they match |
Lessons archive and trial history |
Last promoted bundle |
OPRO |
|
|
Best training score, or the best validation mean of the top |
MWM |
Maintenance and a learner session per eligible block, plus native bootstrap/retry behavior |
World-model adapter plus method/session records |
Current world-model adapter |
ReHarness |
Per instance, one reflection call, one revision call per retrieved reflection (up to k) and one embedding call; no candidate rollouts or validation |
Reflection bank in the adapter; source trajectories and per-instance retrieval and revision records in |
Current adapter: the bank after the last instance |
Meta-Harness Adapter |
One proposer session per round writing k candidates; each candidate is a full rollout of the search set, one step per candidate |
Every candidate’s files, hypothesis, base and per-task scores in |
Best mean on the search set |
ASPIRE |
Per task, one coordinator session and, when it proposes entries, a verifier rollout (a fresh actor on the task’s verifier row) |
The coordinator’s curriculum and changelog notes in |
Current adapter: the library after the last task |
RRSI Adapter |
Per round: the analyst with a digester per rendered trace (a session per turn), m proposer sessions with the critic’s review and repairs, a smoke on the domain’s smoke tasks per screened candidate; each candidate is a full rollout of the evolve set (k rows per task), one step per candidate |
Every candidate’s files and digest, the edit history, the attribution and the frontier in |
The incumbent H_T, admitted above the noise floor and within the cost rule |
use_gate: false in SkillOpt removes the ordinary improvement requirement, not the validation calls.
GEPA’s minibatch criterion is separate from its validation-based parent/final selection.
Use run records to compare actual calls and costs rather than treating every step as equal work.
Decide what the learner may change#
For context-only experiments, start with ADAPTER.md. To learn skills or tool implementations, declare those
paths as learnable in a custom template. Changing evolver.name does not expand those permissions. The
customization tutorial shows how to create a separate template identity.
A text optimizer writes selected text files. A coding-agent Evolver can make more general edits, within the template’s
rules. MWM has its own memory, programs, evidence protocol, and learner contract; choosing it requires a task
integration that supplies those resources. ReHarness writes its reflection bank (reflections/*.json, index.json)
and leaves ADAPTER.md alone; its acting agent must read the adapter through Session.context(query, task).
Meta-Harness Adapter may write any file of the adapter outside .hermit/; what reaches the acting agent besides
ADAPTER.md depends on the agent, and the task information says what it is.
Plan the data and comparison#
Keep test tasks for evaluation. GEPA and SkillOpt additionally require validation tasks for candidate selection.
For an ordered stream, set
data.shuffle: false; choose batch boundaries according to the environment’s timeline.Hold acting model, tools, judge, and evaluation protocol fixed when comparing methods.
Compare against both no adapter and the starting adapter. Their difference measures the template’s initial contribution.
Record learning calls, candidate evaluations, and acting calls. Equal training steps do not imply equal cost.
The framework checks declared needs and file validity. It does not automatically establish learning effectiveness or choose a method for you. Continue with Train an experiment.