Write your own Evolver#

Implement a class with a step method when you want a different way to turn task evidence into adapter edits. The task pipeline, Trainer checks, and version history remain the framework’s responsibility.

A complete offline learner#

The Quick Start’s learner reads the current batch’s judge feedback and writes one supported formatting rule:

"""Learn one supported output rule from scored training feedback, offline."""
import json
from hermit.agent.evolver.base import Evolver, StepContext, StepResult


class FormatEvolver(Evolver):
    def step(self, ctx: StepContext) -> StepResult:
        tasks = ctx.workspace.learning_material / "current" / "tasks"
        records = [json.loads(p.read_text()) for p in sorted(tasks.glob("*.json"))]
        rule = "Output format: uppercase."
        learned = any(r["judge"]["score"] < 1 and rule in r["judge"]["feedback"] for r in records)
        if learned:
            (ctx.workspace.adapter / "ADAPTER.md").write_text("# Output policy\n\n" + rule + "\n")
        summary = "learned uppercase output rule" if learned else "kept current output rule"
        (ctx.workspace.output / "report.md").write_text(summary + "\n")
        return StepResult(status="done", summary=summary)

It is deliberately narrow: it can extract only one rule. A real method can analyze trajectories, generate and test candidates, and select which files to return. The adapter result is the directory left in ctx.workspace.adapter; StepResult reports the call’s status, summary, and externally incurred usage.

Its manifest declares the class and defaults:

description: Deterministic teaching learner; learns one output rule from training feedback.
class: evolver.py:FormatEvolver
needs: []
params: {}

Run and modify a copy#

mkdir -p .demo
cp -R docs/_examples/first_training .demo/my-evolver

In the copied run.yaml, set trainer.output_dir: runs and trainer.run_name: learner-demo. Leave evolver.name: learner pointing to the copied definition, then run:

hermit check --config .demo/my-evolver/run.yaml
hermit train --config .demo/my-evolver/run.yaml
hermit eval --config .demo/my-evolver/run.yaml --adapter final
cat .demo/my-evolver/runs/learner-demo/checkout/ADAPTER.md

Change the learner implementation in that copy and use a new run name for each completed experiment. No tracked task file needs to be edited or reset.

Understand the workspace#

Path

Purpose

adapter

Working adapter files to edit

learning_material/current/

Current batch’s task records and summary

task_info

Learner-facing task contract

state

Persistent method-owned state, saved at completed steps

scratch

Temporary per-step work

output

Reports and artifacts collected with the step

history

Earlier adapter versions and diffs

Store search state in state, not only in Python attributes. Crash recovery constructs a new process. An unchanged adapter is a valid outcome. A failed call keeps the previous adapter; it does not imply rolling back every learner-state update.

Use metered tools#

With a complete llm section in the run config, the run’s model reaches the learner as ctx.agent, a direct Base Agent (Agent layers) metered against the budget:

from hermit.agent.evolver.base import ask

reply = ask(ctx.agent, [{"role": "user", "content": prompt}], max_tokens=4096)
reply["content"], reply["finish_reason"]

The llm tool (ctx.tools["llm"], and the tool endpoint for an external program) calls the same agent. The framework accounts for this usage. Report calls made outside metered tools through StepResult.usage, and avoid double counting.

For candidate tests:

result = ctx.tools["run_agent"](task_ids=["train-1"], adapter=str(candidate_directory))

Candidate directories must be inside the run directory; only training/validation task ids are accepted. Those calls count toward learner usage. They do not expose test evaluation to the learner.

Choose a different execution form#

Form

Use when

Evolver subclass

Your method runs in Python

AgentEvolver

A coding-agent host edits files under your prompts

CommandEvolver

An external program receives workspace context through HA_CONTEXT

Subclass of a built-in method

You need to modify a small piece of an existing implementation

A learner program or session runs in the evolver sandbox: the workspace under /w, its model’s key variable and provider host, no other host unless sandbox.evolver.allowed_hosts names it. Before a reported run, keep sandbox.enforce: required, set web_search explicitly, and read the launch records (runs/<run>/sandbox/learner-*.json): the isolation check passed, and the allowed hosts and passed variables are the ones you expect. On macOS there is no evolver sandbox: sandbox.enforce: required refuses to start, and a run made unconfined on purpose (sandbox.enforce: best_effort or sandbox.enabled: false) does not restrict file reads, which a result must state.

The full extension contract covers setup, bootstrap, finalize, types, session resume, and cancellation. The Evolver component covers status and state semantics. Verify failure handling, valid/invalid edits, persisted state, and usage accounting before using paid model calls.