Write your own Evolver#
Implement a class with a step method when you want a different way to turn task evidence into adapter edits.
The task pipeline, Trainer checks, and version history remain the framework’s responsibility.
A complete offline learner#
The Quick Start’s learner reads the current batch’s judge feedback and writes one supported formatting rule:
"""Learn one supported output rule from scored training feedback, offline."""
import json
from hermit.agent.evolver.base import Evolver, StepContext, StepResult
class FormatEvolver(Evolver):
def step(self, ctx: StepContext) -> StepResult:
tasks = ctx.workspace.learning_material / "current" / "tasks"
records = [json.loads(p.read_text()) for p in sorted(tasks.glob("*.json"))]
rule = "Output format: uppercase."
learned = any(r["judge"]["score"] < 1 and rule in r["judge"]["feedback"] for r in records)
if learned:
(ctx.workspace.adapter / "ADAPTER.md").write_text("# Output policy\n\n" + rule + "\n")
summary = "learned uppercase output rule" if learned else "kept current output rule"
(ctx.workspace.output / "report.md").write_text(summary + "\n")
return StepResult(status="done", summary=summary)
It is deliberately narrow: it can extract only one rule. A real method can analyze trajectories, generate and
test candidates, and select which files to return. The adapter result is the directory left in
ctx.workspace.adapter; StepResult reports the call’s status, summary, and externally incurred usage.
Its manifest declares the class and defaults:
description: Deterministic teaching learner; learns one output rule from training feedback.
class: evolver.py:FormatEvolver
needs: []
params: {}
Run and modify a copy#
mkdir -p .demo
cp -R docs/_examples/first_training .demo/my-evolver
In the copied run.yaml, set trainer.output_dir: runs and trainer.run_name: learner-demo. Leave
evolver.name: learner pointing to the copied definition, then run:
hermit check --config .demo/my-evolver/run.yaml
hermit train --config .demo/my-evolver/run.yaml
hermit eval --config .demo/my-evolver/run.yaml --adapter final
cat .demo/my-evolver/runs/learner-demo/checkout/ADAPTER.md
Change the learner implementation in that copy and use a new run name for each completed experiment. No tracked task file needs to be edited or reset.
Understand the workspace#
Path |
Purpose |
|---|---|
|
Working adapter files to edit |
|
Current batch’s task records and summary |
|
Learner-facing task contract |
|
Persistent method-owned state, saved at completed steps |
|
Temporary per-step work |
|
Reports and artifacts collected with the step |
|
Earlier adapter versions and diffs |
Store search state in state, not only in Python attributes. Crash recovery constructs a new process.
An unchanged adapter is a valid outcome. A failed call keeps the previous adapter; it does not imply rolling
back every learner-state update.
Use metered tools#
With a complete llm section in the run config, the run’s model reaches the learner as ctx.agent, a direct
Base Agent (Agent layers) metered against the budget:
from hermit.agent.evolver.base import ask
reply = ask(ctx.agent, [{"role": "user", "content": prompt}], max_tokens=4096)
reply["content"], reply["finish_reason"]
The llm tool (ctx.tools["llm"], and the tool endpoint for an external program) calls the same agent. The
framework accounts for this usage. Report calls made outside metered tools through
StepResult.usage, and avoid double counting.
For candidate tests:
result = ctx.tools["run_agent"](task_ids=["train-1"], adapter=str(candidate_directory))
Candidate directories must be inside the run directory; only training/validation task ids are accepted. Those calls count toward learner usage. They do not expose test evaluation to the learner.
Choose a different execution form#
Form |
Use when |
|---|---|
|
Your method runs in Python |
|
A coding-agent host edits files under your prompts |
|
An external program receives workspace context through |
Subclass of a built-in method |
You need to modify a small piece of an existing implementation |
A learner program or session runs in the evolver sandbox: the workspace under /w, its model’s key variable and
provider host, no other host unless sandbox.evolver.allowed_hosts names it. Before a reported run, keep
sandbox.enforce: required, set web_search explicitly, and read the launch records
(runs/<run>/sandbox/learner-*.json): the isolation check passed, and the allowed hosts and passed variables are the
ones you expect. On macOS there is no evolver sandbox: sandbox.enforce: required
refuses to start, and a run made unconfined on purpose (sandbox.enforce: best_effort or sandbox.enabled: false) does
not restrict file reads, which a result must state.
The full extension contract covers setup, bootstrap, finalize, types, session resume, and cancellation. The Evolver component covers status and state semantics. Verify failure handling, valid/invalid edits, persisted state, and usage accounting before using paid model calls.