Extend and contribute#

Start with the extension boundary you need. A task integration normally requires no Trainer change.

Change

Entry point

Verify

New task, acting agent, or judge

Bring your task, task contract

No-adapter probe, adapter consumption, split isolation, logs, scoring

New instruction/skill/tool structure

Customize an adapter, adapter metadata

Learnable/fixed rules, checker, exported schemas

New learning method

Evolver tutorial, full contract

Failure handling, state persistence, usage, final selection

Reproduction of a published method

Reproduce a published method

The adapter and the Evolver named, the differences from the authors’ code, the protocol, the first evaluation on the shared settings

New host exporter

hermit/hosts/, hermit/plugins/

Supported exports, refusal/omission behavior, repeat generation

New CLI Base Agent

A CliAgent subclass in CLIS (hermit/agent/base/cli.py), agent layers

Every hook, limits it cannot keep refused, its stream read (reply, steps, usage, served) from a recorded stream

Trainer behavior

Trainer contract, hermit/trainer/

Rollback, resume boundaries, records, budget observations

Repository map#

Location

Responsibility

hermit/cli.py, config.py

Command parsing and config resolution

hermit/agent/

The agent layers: model/ (Layer 1), base/ (Layer 2, the Base Agents), acting/ and evolver/ (Layer 3)

hermit/agent/acting/

The Acting Agent interface and the built-in acting agents

hermit/agent/evolver/

The Evolver interface and the agent Evolver; methods/ holds the built-in learning methods

hermit/task.py

Task records, callable loading, pipeline/judge types

hermit/adapter/

Manifest loading and acting consumption

hermit/adapter/templates/

Built-in adapter seeds and specifications

hermit/trainer/

Training loop, rollouts, material, budgets, recovery

hermit/runtime/

Tool serving, sandboxing, and runtime support

examples/<task>/utils/

Task integrations and benchmark-owned acting/scoring logic

tests/

Regression tests: core/ for the framework, methods/<name>/ for each method

docs/_examples/

Small offline examples included in documentation

Local development#

python -m pip install -e '.[test]'
python -m pytest --changed

--changed runs the test groups the branch touches (tests/README.md): the core group for every change but documentation, a method’s group for its own files, every group for a shared layer.

Optimizer tests need the method’s dependency group (for example python -m pip install -e . --group gepa) and skip without it; sandbox and MWM integration tests can require Linux, learner CLIs, external packages, or environment fixtures. Inspect their prerequisites rather than interpreting every skipped test as coverage.

For a new Evolver, put its tests under tests/methods/<name>/ and add its paths to METHODS in tests/selection.py (tests/core/test_selection.py fails until every method has a group); use an offline task or fake client to test real failure and state behavior without paid model calls. A new or changed test follows the test checklist.

Documentation changes#

python -m pip install -r docs/requirements.txt
bash docs/build.sh
bash docs/serve.sh 8000

The build treats warnings as errors. Link new pages from docs/index.md, use repository-relative literalinclude for code and defaults, and execute walkthrough commands in a fresh run directory.

Every recipe should state its prerequisites, data preparation, model setup, launch command, output locations, evaluation protocol, reproduction limits, and available result cards. Keep observed results distinct from smoke tests and deterministic demonstrations.

For changes to interactive figures, inspect desktop and narrow layouts plus keyboard operation. The existing component contracts are described in Adapter, Evolver, and Trainer.

Proposing a change#

Describe the user-visible behavior, the affected contract, how you verified it, and any environment requirements. Include a small reproducer for failures. Update commands, defaults, and docs together; do not describe a proposed feature as available. The generic service lifecycle proposal is kept separately from supported usage. The sandbox design records how the task, judge and evolver sandboxes are built.