Task and acting pipeline#

A task integration connects the framework to the problems and acting system you want to improve. You supply data, a description, a pipeline, and feedback. Read this page to bring your own agent or benchmark.

Responsibility and boundaries#

The pipeline owns the acting model or agents, their execution loop, environment access, and runtime state. It decides how to deploy the adapter. The Trainer chooses tasks and collects results; the Evolver consumes those results to improve the adapter.

Trainertask + adapter →
← RunOutput
PipelineActing system
Trainertask + output →
← feedback
Scoring function
The same task interface serves training, candidate experiments, and evaluation.

This boundary allows a single agent, a group of agents, or a custom workflow to use the same training interface.

Interface#

The current implementation calls two functions:

run(task: dict, adapter: Path | None) -> RunOutput
score(task: dict, output: RunOutput) -> JudgeResult

run receives one task and an adapter directory. It must make the required exports available to the acting system before running the task. adapter=None means a run without adapter exports, used for baseline comparison. The pipeline should preserve its ordinary behavior in either case.

score evaluates the returned output. It can instead return a dictionary containing score and feedback, or a numeric score. Additional dictionary fields are collected under the normalized result’s extra field.

Scoring is part of collecting learning material. Although the general architecture can use other feedback sources, the current train and eval commands require a configured scoring callable.

Execution result#

Only output is required when constructing RunOutput:

@dataclass
class RunOutput:
    output: Any                          # what the judge scores
    log: Path | None = None              # the host's original log (the trajectory), a file or a directory
    cost_usd: float | None = None        # None = unknown
    wall_s: float = 0.0
    api_speed: dict | None = None        # for example output tokens per second
    messages: list | None = None         # optional normalised message list of the trajectory
    tokens: int | None = None            # input + output tokens, if known
    extra: dict = field(default_factory=dict)

Field

Meaning for the integration

output

The value the scoring function receives. Choose a JSON-serializable task-specific representation.

log

Optional file or directory containing the original execution log. The Trainer copies it into learning material.

messages

Optional normalized trajectory for methods that use model messages.

cost_usd, tokens, wall_s

Usage and timing. Unknown cost/tokens should remain null; zero means known zero.

api_speed, extra

Optional measurements and task-specific metadata.

Concurrent tasks may run against the same adapter path. Treat that adapter as input and keep execution state and logs outside it. Give each execution its own working paths unless sharing is part of the task design. Keep returned log paths available until the Trainer has collected them.

Feedback result#

@dataclass
class JudgeResult:
    score: float
    feedback: str = ""
    extra: dict = field(default_factory=dict)

The scoring function decides what feedback to reveal. Reference answers placed in feedback become visible to the learner. The numeric score can be task-specific; the current eval report counts a task as solved at least once when any repeated score is at least 1.0.

Built-in agents#

A task whose acting agent is a tool-calling model, or a coding-agent CLI, names a built-in agent as its pipeline and writes no loop:

agent:
  name: react
  params:
    model: fireworks/accounts/fireworks/models/glm-5p3-flash   # <backend>/<model id>; or {base_url, model, api_key_env, price}
    prompt: utils/agent/prompt.md      # system prompt; `{context}` marks where ADAPTER.md goes (else appended)
    tools: utils/tools.py              # the task's tools: functions marked with @hermit.agent.acting.tool
    user: "{question}"                 # the first user message from the task's fields
    max_turns: 24
    answer_tag: FINAL_ANSWER           # optional: one more call without tools when the tag is missing

agent.name

Agent

Settings

react

a chat-completions tool-calling loop over the task’s tools and the adapter’s tools

model, prompt, tools, user, max_turns, max_output_tokens, max_tokens, temperature, answer_tag, tool_result_chars, logs

codex, claude-code, pi

Codex, Claude Code or Pi run in a fresh workspace the adapter is deployed into (Pi: no login, no roles; its project’s .pi/ is trusted for the session)

model, login, effort, prompt, files, mounts, prepare, answer, collect, context_query, project_doc_max_bytes, args, env, price, timeout_s, retries, retry_wait_s, workspaces

mini-swe-agent

mini-swe-agent run in a fresh workspace the adapter is deployed into (no login, no roles; tutorial)

model, model_class, config, step_limit, effort, prompt, files, mounts, prepare, answer, collect, context_query, env, price, timeout_s, retries, retry_wait_s, workspaces

opencode

OpenCode run in a fresh workspace the adapter is deployed into (no login, no roles; a model reference only; the deployment’s skills only)

model, effort, prompt, files, mounts, prepare, answer, collect, context_query, args, env, price, timeout_s, retries, retry_wait_s, workspaces

dsh

DeepSeek Harness run in a fresh workspace the adapter is deployed into (no login, no roles; permission_mode sets its own sandbox: read-only, workspace-write, the default, or danger-full-access)

model, provider, permission_mode, image_input, effort, prompt, files, mounts, prepare, answer, collect, context_query, args, env, price, timeout_s, retries, retry_wait_s, workspaces

hermes

Hermes Agent run in a fresh workspace the adapter is deployed into (no login, no roles, no web search; a model reference only; max_turns caps its model requests a turn, Hermes’s 90 by default)

model, provider, max_turns, effort, prompt, files, mounts, prepare, answer, collect, context_query, args, env, price, timeout_s, retries, retry_wait_s, workspaces

A coding-agent CLI as the acting agent#

codex, claude-code, pi, mini-swe-agent, opencode, dsh and hermes are the framework’s one way to act with a coding-agent CLI: a task gives what is its own, in configuration and, where its workspace is built in code, in functions it names (file.py:function); the agent does the rest. OfficeQA’s Codex runs are one such configuration (examples/officeqa/utils/task.py):

agent:
  name: codex
  params:
    model: fireworks/accounts/fireworks/models/glm-5p3-flash   # or the CLI's model name with `login: chatgpt` (Codex)
                                                               # or `login: claude` (Claude Code)
    effort: medium
    prompt: ../utils/prompt.md            # a template of the task's fields, or file.py:function (task) -> prompt
    prepare: ../utils/task.py:prepare     # (task, workspace[, params, base]) -> mounts: writes the task's files
    answer: ../utils/task.py:answer       # (last message, task) -> the output; or a pattern, the last match
    collect: ../utils/task.py:collect     # (task, workspace, log) -> facts: the task's own artifacts into the log
    project_doc_max_bytes: 65536          # Codex's cap on AGENTS.md; a deployment past it is refused
    timeout_s: 3600
    workspaces: ~/.cache/hermit/officeqa-codex     # outside any checkout: Codex reads a repository's instructions
environment:                              # the acting agent's sandbox: here a Docker container per session
  image: "officeqa-codex:0.162.0"
  container: {network: oq-sandbox, proxy: "http://oq-egress:3128"}

What the agent does for every task:

  • Deploy. The adapter is deployed as hermit deploy deploys it (Acting interface): the whole adapter under hermit/<adapter id>/ but its internal files, each export connected to the CLI’s own place, and the context naming the rest. Its tools are served by an MCP server on this machine; a container session, which cannot start one, gets them from the session’s own tool server, which the container reaches, and so does an OpenCode session (OpenCode waits only 2 s for a server it starts) and a Hermes Agent session (Hermes reads MCP servers from its home’s config only, and is given the deployment’s skills folder). An adapter’s context_tool is rendered for each task (the field context_query, else the task’s question or prompt), deployed into the context and recorded in RunOutput.extra["context_tool"]; so is the context of any adapter whose template is not plugin-portable, which hermit deploy alone refuses. The session calls the adapter’s tools without approvals and may start its sub-agents. An adapter with memory/NOTES.md (the continual-harness template) is told so in the prompt, and the notes the session leaves come back in the log (log/memory/). The context must fit project_doc_max_bytes (Codex reads no further): a longer one is refused. So is one too long for DeepSeek Harness, which gives the model at most 65,536 bytes of a project’s instructions. Hermes Agent reads the context from .hermes.md and takes the prompt as one command-line argument, under 128 KiB (120,000 bytes): a longer prompt is refused, not cut. Hermes’s prompt-injection scan replaces a context file it matches (benign rules can match) with a notice only the model sees: that session is an error, naming the file.

  • Isolate. With environment.image, the CLI runs in a container of its own, started from a worker on the host that runs the adapter’s code the deployment renders or serves in the strict sandbox: the workspace and the session’s home read-write, the mounts read-only at their targets, an internal network whose only way out is an egress proxy to the provider’s hosts, and the CLI’s own sandbox inside for the commands it runs (Codex’s workspace-write; Claude Code’s, with no network and writes to the workspace and /tmp only). docker/codex/ builds Codex’s image and the proxy, docker/claude/ Claude Code’s image; environment.container (or HERMIT_CODEX_NETWORK, HERMIT_CODEX_PROXY, and HERMIT_CLAUDE_NETWORK, HERMIT_CLAUDE_PROXY for Claude Code) names the network and the proxy. The agent checks the image and the network before its first session (for Claude Code, also the CLI in the image and that its sandbox starts there). Without an image the agent runs in the task sandbox (bubblewrap), where a mount inside the workspace is a link for Codex; for Claude Code, DeepSeek Harness and Hermes Agent it is bound there by a nested bubblewrap, a real read-only directory as in a container (a search that walks the tree, grep -r, find, Claude Code’s Glob and Grep, DeepSeek Harness’s glob and grep or Hermes’s search_files, does not follow a link), and linked with a warning where bubblewrap cannot nest. Hermes Agent runs in the task sandbox or with no isolation, not in a container. DeepSeek Harness also confines its commands and file tools in a sandbox of its own, inside the task sandbox (permission_mode). Its default, workspace-write, lets them write only the workspace and /tmp, so not environment.write. Where neither bubblewrap nor Landlock (the Linux kernel’s own way of confining a process) can start there, it refuses every command and the session goes on. danger-full-access turns it off and leaves the task sandbox as the boundary.

  • Sign in. A <backend>/<model> reference binds Codex to that backend’s key; login: chatgpt uses the user’s own ChatGPT login (codex login; model is then a Codex model name, the cost unknown), from one shared copy (~/.cache/hermit/codex-login/, outside every run’s directory) that each session’s refreshed token is written back to, the task sandbox’s sessions included. On Claude Code login: claude signs in with the token of claude setup-token in CLAUDE_CODE_OAUTH_TOKEN, passed by name (nothing is copied), and nothing else may pay (Claude subscription). In the task sandbox a reference or the CLI’s own login is required: the CLI’s own sign-in from the user’s home is not copied in (a model that is no reference uses it only where the pipeline runs on the host). Each session has a home of its own: none of the user’s Codex, Claude Code, DeepSeek Harness or Hermes Agent settings apply, and a Codex session’s commands see only Codex’s core variables (shell_env: all passes every variable). A DeepSeek Harness session never gets the caller’s DSH_*, DEEPSEEK_BASE_URL or DEEPSEEK_SEARCH_BASE_URL, even through environment.env: give them in the agent’s env setting. Nor does a Hermes Agent session get the caller’s HERMES_*, TERMINAL_* or TIRITH_*, or another provider’s key (OPENROUTER_API_KEY, …): give them in env. Its auxiliary tasks go to its own model, and it has no fallback provider, no memory, no skill-creation nudges, no title request, no update check and no tirith download. Hermes itself still loads the .env beside its own source (a checkout installed from source) and a machine-wide managed .env, over what Hermit passed: keep keys out of those files for a Hermes that Hermit runs.

  • Web search. web_search must be set, true or false: Codex’s hosted search runs at the provider, past any sandbox or proxy, so it is never left to a default. Claude Code’s WebSearch and WebFetch follow it too, and every Claude Code session is denied the tools whose work outlives it or reaches past it: RemoteTrigger, CronCreate, CronDelete, CronList, ScheduleWakeup, ListAgents and SendMessage (Claude subscription). DeepSeek Harness’s web search and fetch follow web_search too; its search is DeepSeek’s own service, so unless the model is reached through DeepSeek’s API on DEEPSEEK_API_KEY, true gives the session web fetch only. Its fetch runs in the session, not at the provider: in the task sandbox it reaches only what environment.network allows. Hermes Agent has none: web_search must be false (its search and extraction are third-party services that reach past the sandbox).

  • Stop and retry. One session per run, stopped at timeout_s. A session the provider could not serve (429, 5xx, a container that did not start) runs again after retry_wait_s, up to retries; then the run is an error, not a wrong answer. On Claude Code, DeepSeek Harness and Hermes Agent any failed session is an error, with its tokens and log kept and no judge call, and on Claude Code a lost sign-in on a login ends the run, which hermit train resumes. A session past its clock is an attempt with no answer. Hermes Agent runs again only when Hermes classed its last failed request as a provider that could not serve (rate_limit, upstream_rate_limit, overloaded, server_error, timeout, or a 429 or 5xx); at max_turns it asks for a summary and exits 0, and that session is failed (finish_reason length), the summary its reply.

  • Record. The run’s directory holds work/ (the workspace, kept for inspection without its links out), home/ and log/. RunOutput.log is log/: the host’s event stream (agent-codex.jsonl, agent-claude.jsonl, agent-dsh.jsonl, agent-hermes.jsonl), prompt.txt, last_message.txt, deployment.json, and what collect added. The output is what answer returned, with fail_reason when there is no answer; without answer, the last message. In a run, a model key’s value in what the call leaves is scrubbed (<VARIABLE>; the login’s token too, as <CLAUDE_CODE_OAUTH_TOKEN>). RunOutput.extra has the same keys for every task: backend, model, effort, web_search, deploy, isolation, deployed, exit_code, timed_out, attempts, workspace, mounts (each mount’s target inside the workspace, bound or linked), and context_tool and task when there are any; session_failed says why a session failed, which makes the run an error record with no judge call.

The agent records; the Evolver interprets. Every Evolver reads a host’s stream through hermit.agent.base.trajectory (read(record, step_dir) gives the run’s steps: messages, reasoning, commands with their exit codes and output, file changes, tool calls, errors) and renders what its learner needs: SkillOpt rebuilds its Codex trace summary, GEPA a text of the steps. A hook adds to the record, never replaces it, so every Evolver sees the same run.

Every agent has the same interface, the task’s own included:

from hermit.agent.acting import Agent

class MyAgent(Agent):                     # agent.name: agent/agent.py:MyAgent
    def run(self, task, adapter):         # self.params, self.base (the config's directory), self.session(adapter)
        ...
        return RunOutput(output=...)

A plain run(task, adapter) function is accepted too (agent.name: agent/pipeline.py:run); it receives params and base when its signature names them. A task tool is a plain function:

from hermit.agent.acting import tool

@tool("Read lines of one bulletin file.", {"type": "object", "properties": {
      "filename": {"type": "string"}, "start_line": {"type": "integer"}}, "required": ["filename"]})
def read(filename: str, start_line: int = 1) -> dict:
    ...

A function that declares _task receives the task row. The result is JSON; an exception becomes an error message the model reads. The adapter’s own tools are offered beside these and answered through Session. A benchmark that owns its agent loop is an Agent subclass that runs that loop with Session plugged in; examples/clbench_poker/utils/agent/agent.py is one. For an adapter whose manifest names a context_tool, such a loop reads the context with session.context(query=<the instance's input>, task=<the row>) and records session.last_context in RunOutput.extra["context_tool"], as react does, so the Evolver knows what the agent read.

Benchmarks#

A benchmark enters Hermit as a task set in Harbor’s format: that is the one protocol for every task, whether its agent works inside the container (a coding or terminal benchmark, a corpus searched with shell tools) or only answers (a question stream whose state the environment keeps), instead of Docker code or a loop of the benchmark’s own. The format (task.toml, instruction.md, environment/ with a Dockerfile or a prebuilt docker_image, tests/test.sh writing /logs/verifier/reward.txt) is what Terminal-Bench, SWE-bench and the datasets on the Harbor hub share, and the built-in harbor agent makes one rollout one Harbor trial:

agent:
  name: harbor
  concurrency: 8
  params:
    tasks_dir: ../utils/tasks            # the task directories (python -m hermit.harbor.dataset downloads or indexes)
    model: gpt-5.6-luna                  # the acting model; a `<backend>/<model>` reference binds an endpoint and key
    agent_timeout_sec: 3600              # a uniform agent budget; null: each task's own
    jobs_dir: ~/.cache/hermit/harbor/jobs
judge:
  path: hermit.harbor.judge              # the reward the container's verifier wrote

The pipeline of a Harbor task set runs on the host: the task’s container is its sandbox, and the verifier runs there after the agent (Harbor’s [verifier] environment_mode: shared, or a separate container).

Harbor starts the task’s container, installs the acting agent in it (Codex by default: harborside.codex:HermitCodex, Harbor’s Codex with the adapter deployed into the container as hermit deploy lays it out, the AGENTS.md context, skills, roles and hooks, with the whole adapter copied to hermit/<adapter id>/ in the task’s workdir), runs the instruction and then the task’s own verifier; the trial’s reward is the rollout’s output and the judge’s score. A task row (tasks.jsonl, written by python -m hermit.harbor.dataset index <tasks dir> --out tasks.jsonl [--rollouts k]) names its directory as harbor_task and is the task’s name, so a benchmark’s existing split files apply; k rollouts of a task are k rows (<task>#<k>). Harbor lives in its own venv (~/.cache/hermit/harbor-venv, agent.params.harbor names its command): Hermit’s side writes one job config per rollout and runs harbor run as a subprocess, and the container-side half under hermit/harbor/harborside/ imports harbor and the standard library only.

agent.params

Meaning

tasks_dir, jobs_dir, tmpdir, harbor

the task directories; one Harbor job per rollout (outside the checkout); TMPDIR for harbor; Harbor’s command

model, effort, codex_version, codex_login

the acting model (a name, or a backend reference: its endpoint and key variable go to Harbor); Codex’s reasoning effort, npm pin, and whether the ChatGPT login (~/.codex/auth.json) is uploaded when no endpoint is bound

acting_agent

{import_path} or {name} (Harbor’s built-ins: codex, mini-swe-agent, …; they take no adapter), kwargs, env, pythonpath (directories of an example’s own Harbor agents)

agent_timeout_sec, environment, verifier, retry

the agent’s budget; Harbor’s environment (type, import_path, delete, force_build, mounts, kwargs), verifier and retry configs, passed through

docker_cleanup, rmi_threshold_gb, disk_gate_gb, disk_gate_wait_s

images_keep, rmi_task_image or rmi_over_gb (the task’s prebuilt image removed after the trial, only when larger than the threshold); no trial starts while Docker’s data root has less than the gate free

benchmark

a YAML file holding these same keys, shared by every method’s config of the example (the config’s params override it)

stream

a multi-step task’s driver: idle_s (the environment stops after that long without a step), start_timeout_s, turn_wait_s

The rollout’s output is {harbor_task, rollout, step, reward, rewards, status, failure_cause, exception_type, trial, hooks_active}, with status one of ok (a verifier reward), timeout (Harbor’s AgentTimeoutError; the verifier may still have scored), infra (a sandbox, model-provider, verifier or Harbor failure, a killed harbor run, the disk gate, a lost stream) or agent_error; tokens is known for an ok trial only. The log is a curated copy of the trial (result.json, the agent’s outputs and ATIF trajectory.json, the verifier’s output and reward; never the agent’s session transcripts). A usage-limit or authentication failure writes <jobs_dir>/HALT: every rollout returns at once as infra/halted while it exists.

Streams. A task with [[steps]] in its task.toml (steps/<name>/instruction.md, steps/<name>/tests/) is one environment that lives through every step: a benchmark whose next question depends on what the earlier ones left in the environment. dataset index writes one row per step (<task>:<step>, with step, step_index and steps), so the Trainer sees the stream as consecutive tasks and the learner acts between them; the harbor agent starts a driver with the first step (harborside/driver.py, Harbor’s MultiStepTrial fed one step at a time over a socket in the stream’s directory under jobs_dir/streams/), asks it for each step in order (a step whose turn has not come waits for the earlier ones, running in other rollouts of the batch), and stops it after the last step, an aborted step, or stream.idle_s without a request. With a frozen adapter, harbor run on the same task evaluates the whole stream in one trial. Put the stream’s rows in file order with data.shuffle: false. A stream’s driver outlives the rollout that started it; it runs on the host with the pipeline.

Packaging a benchmark. python -m hermit.harbor.dataset download terminal-bench@2.0 --output <dir> fetches a registry dataset; a benchmark of your own is written as task directories by your utils/prepare.py (the judge goes into tests/, uploaded after the agent ran, so the answer is never where the agent can read it; see examples/officeqa/utils/harbor/build_tasks.py). dataset lock records the sha256 of every task directory and dataset check compares; dataset splits writes deterministic train/val/test lists by task. The task set is checked without a model by running it with the scripted agent (acting_agent: {import_path: harborside.scripted:ScriptedAgent, kwargs: {script: "echo 42 > /app/answer.txt"}}): a script that produces the right answer must score 1 and a wrong one 0 (tests/core/test_harbor_smoke.py does this against Docker).

Limits, each a piece still to build rather than a boundary of the protocol: an adapter whose context comes from a context_tool is refused, as the CLI agents refuse it (its per-task context would never reach the agent in the container); the adapter’s tools are not served in the container (no Hermit there; the config’s mcp_servers are dropped with a log line); what an agent writes into the adapter’s copy in the container (a memory file) does not come back; a benchmark whose acting model only answers (CL-Bench’s host, OfficeQA’s direct harness) needs a host-side acting agent whose tools run in the task’s environment, which does not exist yet, so those examples still run their own loops; hermit eval on a task set still needs the training-only config keys.

What the task owner provides#

Input

Purpose

Used by

tasks.jsonl

Task dictionaries with unique IDs.

Pipeline and scoring function.

Split files

One task ID per line for train, optional validation, and test.

Trainer and evaluation.

task-info/

Description of the task, objective, feedback, and execution.

Evolver.

Pipeline callable

Adapter integration and acting execution.

Training, evaluation, and candidate experiments.

Scoring callable

Score and feedback for a result.

The same execution paths.

Optional resources

Environment paths or task tools granted to the learner.

Evolver.

Run configuration

Connect these inputs with a template and Evolver.

Framework commands.

A task dictionary needs a unique id (task_id is also accepted); its other fields belong to your integration. Split IDs must refer to existing tasks. check detects overlapping splits. A missing test split permits training, but eval needs test tasks.

task-info#

Provide a README.md as the learner’s entry point. Explain what an input means, what a correct result looks like, the optimization objective, and how to interpret feedback. Larger tasks can split this into task.md, goal.md, judge.md, and pipeline.md; these are organizational conventions.

Written task rules inform the learner. They are not automatically enforced by the Trainer’s specification check. Requirements that must be enforced need an appropriate task judge or template check.

Interaction with other components#

The Trainer calls the pipeline on each training task, passes its result to the scoring function, and writes the combined record as learning material. The same functions serve test evaluation and candidate runs requested by the Evolver. They should not rely on one particular caller or call order.

The adapter reaches the acting system through its exports. A context-only integration may read text into its prompt. A tool or service adapter needs the corresponding host integration and resources. check does not prove that a host supports every export.

The task environment and learner access are separate. A pipeline may use resources that the Evolver cannot inspect. evolver_access.environment grants the learner additional paths; evolver_access.tools declares capabilities. Neither setting configures the acting pipeline for you.

These choices keep task execution reusable when the template or learning method changes, and keep the pipeline’s original execution available as the adapter=None baseline.

Minimal example#

This complete offline stub loads adapter context and echoes the task input. Its messages demonstrate the exchange; it does not call a model. Replace the echo with your acting system when integrating a real task.

"""Offline pipeline stub for exercising the interface; no model is called."""
from pathlib import Path

from hermit import Session
from hermit.task import RunOutput


def run(task: dict, adapter: Path | None) -> RunOutput:
    context = Session(adapter).context() if adapter is not None else ""
    # Replace this echo with the acting system that consumes context and the task.
    answer = task["prompt"]
    return RunOutput(
        output=answer,
        messages=[
            {"role": "system", "content": context},
            {"role": "user", "content": task["prompt"]},
            {"role": "assistant", "content": answer},
        ],
        cost_usd=0.0,
        tokens=0,
    )

Its scoring function compares the output with the expected string:

"""Scores the offline echo task."""
from hermit.task import JudgeResult, RunOutput


def score(task: dict, output: RunOutput) -> JudgeResult:
    correct = output.output == task["expected"]
    return JudgeResult(score=float(correct), feedback="match" if correct else "mismatch")

The example config binds these functions as:

pipeline:
  name: pipeline.py:run
judge:
  path: judge.py
  name: score

Paths are relative to the config file. agent.name includes the class or function name; judge.path names the file or module and judge.name names the function (default score). agent.params is the agent’s to read.

Run the offline example to exercise these interfaces. For a real model and document environment, see OfficeQA.