Quick Start: train, evaluate, and deploy#

Run a complete training cycle locally with Python and Git. No model account, dataset download, GPU, or agent CLI is needed. The example is deterministic: a teaching agent recognizes one formatting rule, and a scripted Evolver learns that rule from judge feedback. It demonstrates the training contract, not real-model learning performance.

1. Install and verify#

Follow Installation, activate the environment, and run everything below from the repository root:

hermit --help
hermit check --config docs/_examples/first_training/run.yaml

The probe runs one training task without an adapter. Its score should be 0, because the agent echoes lowercase text while the judge requires uppercase output. That task score does not mean the integration check failed.

2. Measure the baseline#

hermit eval --config docs/_examples/first_training/run.yaml --adapter none

The JSON summary should report tasks: 2, mean_score: 0.0, and errors: 0. The held-out inputs differ from the training inputs. Answers are computed by the judge rather than stored in task rows.

3. Train from feedback#

hermit train --config docs/_examples/first_training/run.yaml

When the first training task fails, the learner reads the judge’s feedback and writes Output format: uppercase. into ADAPTER.md. The next rollout uses that changed adapter on a different task.

All inputs and code are under docs/_examples/first_training/:

data:
  tasks: tasks.jsonl
  train: train.txt
  test: test.txt
  step_size: 1
  shuffle: false
task_info: task-info
judge:
  path: judge.py
template: prompt-only@1.0
evolver:
  name: learner
sandbox:
  enabled: false
  network: none
  env: []
trainer:
  run_name: first-training
  output_dir: ../../../.demo/first-training
  epochs: 1
  budget:
    tokens: 1000
    hours: 0.02
  call_timeout_min: 1
agent:
  concurrency: 1
  name: pipeline.py:run

The sandbox is disabled for this local scripted demonstration. It launches no external learner program; sandbox boundaries matter when using other learners.

4. Evaluate the result#

hermit eval --config docs/_examples/first_training/run.yaml --adapter root
hermit eval --config docs/_examples/first_training/run.yaml --adapter final

Expected mean_score: none = 0.0, root = 0.0, final = 1.0, all with zero execution errors. root is the initial template; final is the run’s selected adapter. This gain is constructed by the toy task and its one-rule interpreter. It verifies that feedback, editing, versioning, and evaluation connect correctly.

For a real agent, the pipeline passes adapter context and tools to a model. See First training with a real model for an API-only walkthrough with included data, then OfficeQA for a benchmark integration. See the evaluation guide.

5. Inspect and deploy#

cat .demo/first-training/first-training/checkout/ADAPTER.md
cat .demo/first-training/first-training/checkout/CARD.md
cat .demo/first-training/first-training/eval/final/summary.json
hermit deploy \
  --adapter .demo/first-training/first-training/checkout \
  --workspace .demo/first-training/workspace \
  --host codex
cat .demo/first-training/workspace/AGENTS.md

The generated AGENTS.md contains the rule. deploy writes a workspace; starting a host is a separate action. For Claude Code, use --host claude-code and a different workspace directory.

Run it again#

An unfinished run resumes automatically. For an independent experiment, choose a new name on every command:

hermit train --config docs/_examples/first_training/run.yaml trainer.run_name=first-training-02
hermit eval --config docs/_examples/first_training/run.yaml --adapter final trainer.run_name=first-training-02

That experiment’s outputs are under .demo/first-training/first-training-02/. See Run records and recovery for completed runs and branching.

Choose your next step#