Reproduce a published method#
A reproduction brings a published learning method into the framework as a template and an Evolver, runs it on one of the examples under the settings the other methods use, and documents how it differs from the authors’ own code. This page is the contribution guide for that work: what to settle before writing code, what the pull request must contain, and in which order to verify. It is written from the reproductions already in the repository (GEPA, SkillOpt, Harness², Continual Harness, ReHarness, Meta-Harness Adapter).
1. What the adapter is and what the Evolver is#
State, before writing code, which part of the method is the adapter (the learned object: the files the acting agent reads or runs) and which is the Evolver (how evidence becomes an update), and name the template and the Evolver the reproduction adds.
The template declares the method’s levers as learnable paths and must expose them in the form the method uses: if the method’s skills are executable, the template carries them as files the acting agent can run (for Codex, a
SKILL.mdwith ascripts/directory), not as prose about skills. A method whose harness is more than one text file does not fitprompt-only.Use an existing template where one fits. A new template says which existing ones were considered and why none fits (their structure, their fixed paths, what reaches the acting agent).
Draw the Evolver’s step as a flow chart: what it reads, what it calls (the learner, the tools, the rollouts), what it writes, and how the final adapter is chosen. The method page carries it (Meta-Harness Adapter has one).
When the framework narrows what the method may change (the acting agent is fixed, only the adapter changes), the method’s name says so, and the method page states the difference from the paper in its first paragraph.
Method implementation belongs in
hermit/adapter/templates/<name>/,hermit/agent/evolver/methods/<name>/and an example underexamples/<task>/<name>/. Dependency registration inpyproject.tomlandhermit/agent/evolver/dependencies.pyfollows the instructions below. Changes to shared framework behavior need a separate discussion and pull request, which lands before the reproduction.Task-side adaptations of an acting agent (for example, deploying a memory file into a Codex workspace and collecting it afterwards) are application code under
examples/and are allowed, listed separately in the pull request.Where the authors publish their code, use it: pin it to a Git commit in a dependency group of
pyproject.toml([dependency-groups]), add the group toall, add the same requirements toMETHOD_REQUIREMENTSinhermit/agent/evolver/dependencies.pyso the missing-dependency error names them, and call the code at its seam, asharness2_upstreamdoes. When the code cannot be used as is, say so and name, per component, which upstream file and function the reimplementation follows.
3. The evaluation protocol#
Say which protocol the method is evaluated under, and why, because the two protocols answer different questions:
Train / test split. The Evolver learns on the training split (with or without validation) and the final adapter is evaluated frozen on the test split. GEPA and SkillOpt are evaluated this way; so is Continual Harness in its
run_codex.yaml.Streaming. The test tasks arrive in one order; each task is answered by the current adapter before the Evolver learns from it, the adapter is carried to the next task, and the stream’s own answers are the result. Harness² is evaluated this way, and Continual Harness in its
stream_codex.yaml; the authors’ own setting of Continual Harness is streaming (online, reset-free, no held-out tasks).
In the streaming protocol the learner may see each task only after answering it, and the pull request says what the learner sees of it (with the judge’s feedback the stream is online supervision; without it, label-free).
4. The first evaluation’s settings#
We recommend running at least one experiment with GLM 5.3 Flash
(fireworks/accounts/fireworks/models/glm-5p3-flash) in both roles, acting and learning: it is cheap enough to
repeat, and it puts the method on the rows the other methods already have.
The first evaluation of a reproduced method runs on the row of settings the other methods already use on the same example, so that it joins their table. For OfficeQA, as of 2026-10-24:
Setting |
Value |
|---|---|
Acting model |
GLM 5.3 Flash, reasoning effort medium |
Learning model |
GLM 5.3 Flash, reasoning effort medium |
Harness |
Codex in a container per session ( |
Starting adapter |
SkillOpt’s initial OfficeQA skill |
Data |
the paper’s 50 / 24 / 172 split, or the seed-0 stream of the 172 test questions |
Evaluation |
the 172 test questions, one pass; exact match |
Baselines on the row |
no adapter, the starting adapter, the methods already measured |
State every deviation from the row (a session wall clock that cannot be matched, a harness the task’s own code fixes) in the table header. Record with the result: the framework commit, the complete config, the model identifiers and effort, the Codex version and image, the concurrency, the dataset revision, the spending and the time, and the standard error a one-pass evaluation carries (about 3.7 points at 172 questions). Report paired differences against the starting adapter, not only accuracies.
5. Verification, in order#
Unit tests with a scripted model: every pass or stage of the Evolver, its rollback path, and the template check on the template root, a seed and a broken tree.
check_specon the template root andhermit checkon the example’s configs.A lite smoke (a few tasks, a budget cap), after which the trajectories are read, not only the scores: each lever the method claims (a skill opened, a helper run, a sub-agent spawned, a memory note written and read back) is seen in a session, and a failure is traced to a capability or to the answer’s form.
The first evaluation under section 4, with its run record.
A method page under
docs/methods/: setup, the step, the parameters (aliteralincludeofevolver.yaml), what the reproduction is not; a row indocs/methods/index.md, the example’sREADME.md, and the results in the example’s recipe.
6. The reproduction’s pull request#
These rules apply to a pull request that reproduces a published method; other pull requests follow CONTRIBUTING.md alone.
Rebase on the current
mainbefore opening it and after any later change tomain; resolve conflicts on the branch; run the test suite after the rebase.Open it as a draft until the first evaluation has run and its row is in the description.
The description carries the table of section 2, the protocol of section 3, the settings and result of section 4, the list of application-side changes, and what is left out. No attribution lines.
Complete the general PR checklist and method reproduction checklist. Give brief answers with links to the supporting files, runs and records.
Follow the repository’s PR template and prepare-pr workflow to place the answers under
Change,ValidationandReview. Answer overlapping items once and reference existing explanations rather than repeating them.