Run records and recovery#

A run is identified by its config’s trainer.run_name and output directory. Relative configuration paths resolve against the config file’s directory. A shared adapter tree stores versions for runs from the same template.

Find the evidence#

Path	Use
`<output_dir>/<run_name>/steps.jsonl`	Completed steps, rollout scores, learner status, usage, and commit references
`<run>/learning_material/step_NNNN/`	Inputs, outputs, feedback, and available trajectories supplied to the learner
`<run>/step_NNNN/`	Saved step records and learner-state snapshots
`<run>/checkout/ADAPTER.md`	Current or finalized instructions
`<run>/checkout/CARD.md`	Provenance and recorded final evaluation
`<run>/summary.json`	Final selection, stop reason, and checkpoints
`<run>/eval/<selector>/`	Separate evaluation summaries and per-attempt results
`<output_dir>/trees/<template-id>/`	Default Git tree location; configurable with `trainer.tree_dir`

For the Quick Start:

cat .demo/first-training/first-training/steps.jsonl
cat .demo/first-training/first-training/summary.json
git -C .demo/first-training/trees/prompt-only log --all --oneline -8

The step score comes from rollouts with the pre-update adapter. Check the learner status, resulting commit, and later held-out evaluation to understand an update. Read the step’s learning material to verify exactly what feedback the learner received.

Watch a running experiment#

Using the real-model walkthrough’s paths:

tail -f .demo/first-model/runs/first-model/steps.jsonl

Each line appears after a complete step, not after every token or individual rollout. Press Ctrl-C to stop following the file. It may not exist during startup or while the first batch/learner call is still running. Use the training terminal or its redirected process log to inspect that interval.

After step 1 completes:

cat .demo/first-model/runs/first-model/learning_material/step_0001/tasks/train-001.json
cat .demo/first-model/runs/first-model/step_0001/output/report.md
git -C .demo/first-model/runs/trees/prompt-only log --all --oneline -8

For an accepted update, compare its pre-update parent and resulting commit directly from its record:

python -c 'import json, subprocess; from pathlib import Path; r=json.loads(Path(".demo/first-model/runs/first-model/steps.jsonl").read_text().splitlines()[0]); subprocess.run(["git", "-C", ".demo/first-model/runs/trees/prompt-only", "diff", r["parent"], r["commit"], "--", "ADAPTER.md"], check=True)'

For multi-file adapters, remove the trailing file filter to see every changed path.

Read a step record#

A reduced version of the offline tutorial’s actual first-step record, with unrelated fields omitted:

{
  "step": 1,
  "epoch": 1,
  "tasks": ["train-1"],
  "rollout": {"mean_score": 0.0, "scores": {"train-1": 0.0}},
  "evolver": {"status": "done", "summary": "learned uppercase output rule"},
  "status": "done",
  "changed": true,
  "check": {"passed": true, "issues": []},
  "repaired": []
}

The rollout score describes the adapter before the update. changed reports whether the resulting commit differs from parent; inspect check and repaired to understand validation and repair. evolver.status: done by itself does not assert that files changed or that performance improved.

Observation	Interpretation / next action
`done`, `changed: false`	A valid no-op, skipped reflection, rejected internal candidate, or unusable rewrite can explain it; read the method report
`failed` or `out_of_budget`	The previous adapter is kept; inspect errors and usage before assuming the whole run stopped
`repaired` is nonempty	The Trainer corrected invalid changes; inspect the final accepted diff rather than only the proposed edit
No new step line yet	Could be active work or an incomplete step; check the process output and learner logs
`summary.json` appears	The run reached finalization; inspect `stop_reason` and `finalize`, not just the presence of a final tag

config.yaml and setup files appear at run preparation. Step output/state snapshots appear at completed boundaries. summary.json and the final card are completion artifacts; eval/ appears only when evaluation is run separately. A budget stop may occur before the next step is committed, so the last line in steps.jsonl does not necessarily account for all in-flight work. The final summary reports the stop reason and spending known to that process; a resumed process restores accounting from completed records.

Resume an interrupted run#

hermit train --config run.yaml

The default trainer.resume_mode: auto restores an unfinished run from its last completed step, including adapter, learner state, recorded progress, and budget accounting. Work after that boundary can be repeated; external pipeline effects and provider charges are not undone.

Keep the config, data, code, model settings, and external environment compatible with the original run. Recovery does not recreate a benchmark service’s external state unless the task integration does so.

A completed run has summary.json and refuses another train with the same name. Choose a new run name. trainer.resume_mode=disable instead starts over after renaming the previous run directory and branch; use it only when a restart is intended.

Branch an experiment from an adapter#

hermit train --config run.yaml trainer.run_name=experiment-02 start_from=experiment-01/final

The referenced tag must exist in the configured template tree. A compatible adapter directory also works. start_from initializes adapter files for a new run; it does not restore the previous learner state. Selecting a recovery step/tag through resume_mode is not implemented.

Distinguish three kinds of persistence#

Mechanism	What it preserves
Trainer crash recovery	Last completed step’s adapter, progress, state, and accounting
Evolver `state/`	Method-owned search candidates, experience, or other serialized state
`evolver.resume`	A learner host session handle, only when the method supports it

Python instance attributes alone do not survive a process restart. See Trainer lifecycle and Evolver state for the precise contracts.