Task and acting pipeline#
A task integration connects the framework to the problems and acting system you want to improve. You supply data, a description, a pipeline, and feedback. Read this page to bring your own agent or benchmark.
Responsibility and boundaries#
The pipeline owns the acting model or agents, their execution loop, environment access, and runtime state. It decides how to deploy the adapter. The Trainer chooses tasks and collects results; the Evolver consumes those results to improve the adapter.
← RunOutputPipelineActing system
← feedbackScoring function
This boundary allows a single agent, a group of agents, or a custom workflow to use the same training interface.
Interface#
The current implementation calls two functions:
run(task: dict, adapter: Path | None) -> RunOutput
score(task: dict, output: RunOutput) -> JudgeResult
run receives one task and an adapter directory. It must make the required exports available to the acting system
before running the task. adapter=None means a run without adapter exports, used for baseline comparison.
The pipeline should preserve its ordinary behavior in either case.
score evaluates the returned output. It can instead return a dictionary containing score and feedback, or a
numeric score. Additional dictionary fields are collected under the normalized result’s extra field.
Scoring is part of collecting learning material. Although the general architecture can use other feedback sources,
the current train and eval commands require a configured scoring callable.
Execution result#
Only output is required when constructing RunOutput:
@dataclass
class RunOutput:
output: Any # what the judge scores
log: Path | None = None # the host's original log (the trajectory), a file or a directory
cost_usd: float | None = None # None = unknown
wall_s: float = 0.0
api_speed: dict | None = None # for example output tokens per second
messages: list | None = None # optional normalised message list of the trajectory
tokens: int | None = None # input + output tokens, if known
extra: dict = field(default_factory=dict)
Field |
Meaning for the integration |
|---|---|
|
The value the scoring function receives. Choose a JSON-serializable task-specific representation. |
|
Optional file or directory containing the original execution log. The Trainer copies it into learning material. |
|
Optional normalized trajectory for methods that use model messages. |
|
Usage and timing. Unknown cost/tokens should remain null; zero means known zero. |
|
Optional measurements and task-specific metadata. |
Concurrent tasks may run against the same adapter path. Treat that adapter as input and keep execution state and logs outside it. Give each execution its own working paths unless sharing is part of the task design. Keep returned log paths available until the Trainer has collected them.
Feedback result#
@dataclass
class JudgeResult:
score: float
feedback: str = ""
extra: dict = field(default_factory=dict)
The scoring function decides what feedback to reveal. Reference answers placed in feedback become visible to the
learner. The numeric score can be task-specific; the current eval report counts a task as solved at least once
when any repeated score is at least 1.0.
Built-in agents#
A task whose acting agent is a tool-calling model, or a coding-agent CLI, names a built-in agent as its pipeline and writes no loop:
agent:
name: react
params:
model: fireworks/accounts/fireworks/models/glm-5p3-flash # <backend>/<model id>; or {base_url, model, api_key_env, price}
prompt: utils/agent/prompt.md # system prompt; `{context}` marks where ADAPTER.md goes (else appended)
tools: utils/tools.py # the task's tools: functions marked with @hermit.agent.acting.tool
user: "{question}" # the first user message from the task's fields
max_turns: 24
answer_tag: FINAL_ANSWER # optional: one more call without tools when the tag is missing
|
Agent |
Settings |
|---|---|---|
|
a chat-completions tool-calling loop over the task’s tools and the adapter’s tools |
|
|
Codex, Claude Code or Pi run in a fresh workspace the adapter is deployed into (Pi: no |
|
|
mini-swe-agent run in a fresh workspace the adapter is deployed into (no |
|
|
OpenCode run in a fresh workspace the adapter is deployed into (no |
|
|
DeepSeek Harness run in a fresh workspace the adapter is deployed into (no |
|
|
Hermes Agent run in a fresh workspace the adapter is deployed into (no |
|
A coding-agent CLI as the acting agent#
codex, claude-code, pi, mini-swe-agent, opencode, dsh and hermes are the framework’s one way to act with a coding-agent CLI: a task gives what is its own,
in configuration and, where its workspace is built in code, in functions it names (file.py:function); the agent
does the rest. OfficeQA’s Codex runs are one such configuration (examples/officeqa/utils/task.py):
agent:
name: codex
params:
model: fireworks/accounts/fireworks/models/glm-5p3-flash # or the CLI's model name with `login: chatgpt` (Codex)
# or `login: claude` (Claude Code)
effort: medium
prompt: ../utils/prompt.md # a template of the task's fields, or file.py:function (task) -> prompt
prepare: ../utils/task.py:prepare # (task, workspace[, params, base]) -> mounts: writes the task's files
answer: ../utils/task.py:answer # (last message, task) -> the output; or a pattern, the last match
collect: ../utils/task.py:collect # (task, workspace, log) -> facts: the task's own artifacts into the log
project_doc_max_bytes: 65536 # Codex's cap on AGENTS.md; a deployment past it is refused
timeout_s: 3600
workspaces: ~/.cache/hermit/officeqa-codex # outside any checkout: Codex reads a repository's instructions
environment: # the acting agent's sandbox: here a Docker container per session
image: "officeqa-codex:0.162.0"
container: {network: oq-sandbox, proxy: "http://oq-egress:3128"}
What the agent does for every task:
Deploy. The adapter is deployed as
hermit deploydeploys it (Acting interface): the whole adapter underhermit/<adapter id>/but itsinternalfiles, each export connected to the CLI’s own place, and the context naming the rest. Its tools are served by an MCP server on this machine; a container session, which cannot start one, gets them from the session’s own tool server, which the container reaches, and so does an OpenCode session (OpenCode waits only 2 s for a server it starts) and a Hermes Agent session (Hermes reads MCP servers from its home’s config only, and is given the deployment’s skills folder). An adapter’scontext_toolis rendered for each task (the fieldcontext_query, else the task’s question or prompt), deployed into the context and recorded inRunOutput.extra["context_tool"]; so is the context of any adapter whose template is notplugin-portable, whichhermit deployalone refuses. The session calls the adapter’s tools without approvals and may start its sub-agents. An adapter withmemory/NOTES.md(the continual-harness template) is told so in the prompt, and the notes the session leaves come back in the log (log/memory/). The context must fitproject_doc_max_bytes(Codex reads no further): a longer one is refused. So is one too long for DeepSeek Harness, which gives the model at most 65,536 bytes of a project’s instructions. Hermes Agent reads the context from.hermes.mdand takes the prompt as one command-line argument, under 128 KiB (120,000 bytes): a longer prompt is refused, not cut. Hermes’s prompt-injection scan replaces a context file it matches (benign rules can match) with a notice only the model sees: that session is an error, naming the file.Isolate. With
environment.image, the CLI runs in a container of its own, started from a worker on the host that runs the adapter’s code the deployment renders or serves in the strict sandbox: the workspace and the session’s home read-write, the mounts read-only at their targets, an internal network whose only way out is an egress proxy to the provider’s hosts, and the CLI’s own sandbox inside for the commands it runs (Codex’s workspace-write; Claude Code’s, with no network and writes to the workspace and/tmponly).docker/codex/builds Codex’s image and the proxy,docker/claude/Claude Code’s image;environment.container(orHERMIT_CODEX_NETWORK,HERMIT_CODEX_PROXY, andHERMIT_CLAUDE_NETWORK,HERMIT_CLAUDE_PROXYfor Claude Code) names the network and the proxy. The agent checks the image and the network before its first session (for Claude Code, also the CLI in the image and that its sandbox starts there). Without an image the agent runs in the task sandbox (bubblewrap), where a mount inside the workspace is a link for Codex; for Claude Code, DeepSeek Harness and Hermes Agent it is bound there by a nested bubblewrap, a real read-only directory as in a container (a search that walks the tree,grep -r,find, Claude Code’s Glob and Grep, DeepSeek Harness’s glob and grep or Hermes’ssearch_files, does not follow a link), and linked with a warning where bubblewrap cannot nest. Hermes Agent runs in the task sandbox or with no isolation, not in a container. DeepSeek Harness also confines its commands and file tools in a sandbox of its own, inside the task sandbox (permission_mode). Its default,workspace-write, lets them write only the workspace and/tmp, so notenvironment.write. Where neither bubblewrap nor Landlock (the Linux kernel’s own way of confining a process) can start there, it refuses every command and the session goes on.danger-full-accessturns it off and leaves the task sandbox as the boundary.Sign in. A
<backend>/<model>reference binds Codex to that backend’s key;login: chatgptuses the user’s own ChatGPT login (codex login;modelis then a Codex model name, the cost unknown), from one shared copy (~/.cache/hermit/codex-login/, outside every run’s directory) that each session’s refreshed token is written back to, the task sandbox’s sessions included. On Claude Codelogin: claudesigns in with the token ofclaude setup-tokeninCLAUDE_CODE_OAUTH_TOKEN, passed by name (nothing is copied), and nothing else may pay (Claude subscription). In the task sandbox a reference or the CLI’s own login is required: the CLI’s own sign-in from the user’s home is not copied in (a model that is no reference uses it only where the pipeline runs on the host). Each session has a home of its own: none of the user’s Codex, Claude Code, DeepSeek Harness or Hermes Agent settings apply, and a Codex session’s commands see only Codex’s core variables (shell_env: allpasses every variable). A DeepSeek Harness session never gets the caller’sDSH_*,DEEPSEEK_BASE_URLorDEEPSEEK_SEARCH_BASE_URL, even throughenvironment.env: give them in the agent’senvsetting. Nor does a Hermes Agent session get the caller’sHERMES_*,TERMINAL_*orTIRITH_*, or another provider’s key (OPENROUTER_API_KEY, …): give them inenv. Its auxiliary tasks go to its own model, and it has no fallback provider, no memory, no skill-creation nudges, no title request, no update check and no tirith download. Hermes itself still loads the.envbeside its own source (a checkout installed from source) and a machine-wide managed.env, over what Hermit passed: keep keys out of those files for a Hermes that Hermit runs.Web search.
web_searchmust be set,trueorfalse: Codex’s hosted search runs at the provider, past any sandbox or proxy, so it is never left to a default. Claude Code’sWebSearchandWebFetchfollow it too, and every Claude Code session is denied the tools whose work outlives it or reaches past it:RemoteTrigger,CronCreate,CronDelete,CronList,ScheduleWakeup,ListAgentsandSendMessage(Claude subscription). DeepSeek Harness’s web search and fetch followweb_searchtoo; its search is DeepSeek’s own service, so unless the model is reached through DeepSeek’s API onDEEPSEEK_API_KEY,truegives the session web fetch only. Its fetch runs in the session, not at the provider: in the task sandbox it reaches only whatenvironment.networkallows. Hermes Agent has none:web_searchmust befalse(its search and extraction are third-party services that reach past the sandbox).Stop and retry. One session per run, stopped at
timeout_s. A session the provider could not serve (429, 5xx, a container that did not start) runs again afterretry_wait_s, up toretries; then the run is an error, not a wrong answer. On Claude Code, DeepSeek Harness and Hermes Agent any failed session is an error, with its tokens and log kept and no judge call, and on Claude Code a lost sign-in on a login ends the run, whichhermit trainresumes. A session past its clock is an attempt with no answer. Hermes Agent runs again only when Hermes classed its last failed request as a provider that could not serve (rate_limit,upstream_rate_limit,overloaded,server_error,timeout, or a 429 or 5xx); atmax_turnsit asks for a summary and exits 0, and that session is failed (finish_reasonlength), the summary its reply.Record. The run’s directory holds
work/(the workspace, kept for inspection without its links out),home/andlog/.RunOutput.logislog/: the host’s event stream (agent-codex.jsonl,agent-claude.jsonl,agent-dsh.jsonl,agent-hermes.jsonl),prompt.txt,last_message.txt,deployment.json, and whatcollectadded. The output is whatanswerreturned, withfail_reasonwhen there is no answer; withoutanswer, the last message. In a run, a model key’s value in what the call leaves is scrubbed (<VARIABLE>; the login’s token too, as<CLAUDE_CODE_OAUTH_TOKEN>).RunOutput.extrahas the same keys for every task:backend,model,effort,web_search,deploy,isolation,deployed,exit_code,timed_out,attempts,workspace,mounts(each mount’s target inside the workspace,boundorlinked), andcontext_toolandtaskwhen there are any;session_failedsays why a session failed, which makes the run an error record with no judge call.
The agent records; the Evolver interprets. Every Evolver reads a host’s stream through hermit.agent.base.trajectory
(read(record, step_dir) gives the run’s steps: messages, reasoning, commands with their exit codes and output,
file changes, tool calls, errors) and renders what its learner needs: SkillOpt rebuilds its Codex trace summary,
GEPA a text of the steps. A hook adds to the record, never replaces it, so every Evolver sees the same run.
Every agent has the same interface, the task’s own included:
from hermit.agent.acting import Agent
class MyAgent(Agent): # agent.name: agent/agent.py:MyAgent
def run(self, task, adapter): # self.params, self.base (the config's directory), self.session(adapter)
...
return RunOutput(output=...)
A plain run(task, adapter) function is accepted too (agent.name: agent/pipeline.py:run); it receives
params and base when its signature names them. A task tool is a plain function:
from hermit.agent.acting import tool
@tool("Read lines of one bulletin file.", {"type": "object", "properties": {
"filename": {"type": "string"}, "start_line": {"type": "integer"}}, "required": ["filename"]})
def read(filename: str, start_line: int = 1) -> dict:
...
A function that declares _task receives the task row. The result is JSON; an exception becomes an error
message the model reads. The adapter’s own tools are offered beside these and answered through
Session. A benchmark that owns its agent loop is an Agent subclass
that runs that loop with Session plugged in; examples/clbench_poker/utils/agent/agent.py is one. For an adapter
whose manifest names a context_tool, such a loop reads the context with
session.context(query=<the instance's input>, task=<the row>) and records session.last_context in
RunOutput.extra["context_tool"], as react does, so the Evolver knows what the agent read.
Benchmarks#
A benchmark enters Hermit as a task set in Harbor’s format: that is the
one protocol for every task, whether its agent works inside the container (a coding or terminal benchmark, a corpus
searched with shell tools) or only answers (a question stream whose state the environment keeps), instead of Docker
code or a loop of the benchmark’s own. The format (task.toml,
instruction.md, environment/ with a Dockerfile or a prebuilt docker_image, tests/test.sh writing
/logs/verifier/reward.txt) is what Terminal-Bench, SWE-bench and the datasets on the Harbor hub share, and the
built-in harbor agent makes one rollout one Harbor trial:
agent:
name: harbor
concurrency: 8
params:
tasks_dir: ../utils/tasks # the task directories (python -m hermit.harbor.dataset downloads or indexes)
model: gpt-5.6-luna # the acting model; a `<backend>/<model>` reference binds an endpoint and key
agent_timeout_sec: 3600 # a uniform agent budget; null: each task's own
jobs_dir: ~/.cache/hermit/harbor/jobs
judge:
path: hermit.harbor.judge # the reward the container's verifier wrote
The pipeline of a Harbor task set runs on the host: the task’s container is its sandbox, and the verifier runs there
after the agent (Harbor’s [verifier] environment_mode: shared, or a separate container).
Harbor starts the task’s container, installs the acting agent in it (Codex by default: harborside.codex:HermitCodex,
Harbor’s Codex with the adapter deployed into the container as hermit deploy lays it out, the AGENTS.md context,
skills, roles and hooks, with the whole adapter copied to hermit/<adapter id>/ in the task’s workdir), runs the
instruction and then the task’s own verifier; the trial’s reward is the rollout’s output and the judge’s score. A
task row (tasks.jsonl, written by python -m hermit.harbor.dataset index <tasks dir> --out tasks.jsonl [--rollouts k]) names its directory as harbor_task and is the task’s name, so a benchmark’s existing split files
apply; k rollouts of a task are k rows (<task>#<k>). Harbor
lives in its own venv (~/.cache/hermit/harbor-venv, agent.params.harbor names its command): Hermit’s side writes
one job config per rollout and runs harbor run as a subprocess, and the container-side half under
hermit/harbor/harborside/ imports harbor and the standard library only.
|
Meaning |
|---|---|
|
the task directories; one Harbor job per rollout (outside the checkout); TMPDIR for harbor; Harbor’s command |
|
the acting model (a name, or a backend reference: its endpoint and key variable go to Harbor); Codex’s reasoning effort, npm pin, and whether the ChatGPT login ( |
|
|
|
the agent’s budget; Harbor’s environment ( |
|
|
|
a YAML file holding these same keys, shared by every method’s config of the example (the config’s params override it) |
|
a multi-step task’s driver: |
The rollout’s output is {harbor_task, rollout, step, reward, rewards, status, failure_cause, exception_type, trial, hooks_active}, with status one of ok (a verifier reward), timeout (Harbor’s AgentTimeoutError; the verifier
may still have scored), infra (a sandbox, model-provider, verifier or Harbor failure, a killed harbor run, the
disk gate, a lost stream) or agent_error; tokens is known for an ok trial only. The log is a curated copy of
the trial (result.json, the agent’s outputs and ATIF trajectory.json, the verifier’s output and reward; never the
agent’s session transcripts). A usage-limit or authentication failure writes <jobs_dir>/HALT: every rollout returns
at once as infra/halted while it exists.
Streams. A task with [[steps]] in its task.toml (steps/<name>/instruction.md, steps/<name>/tests/) is one
environment that lives through every step: a benchmark whose next question depends on what the earlier ones left
in the environment. dataset index writes one row per step (<task>:<step>, with step, step_index and
steps), so the Trainer sees the stream as consecutive tasks and the learner acts between them; the harbor agent
starts a driver with the first step (harborside/driver.py, Harbor’s MultiStepTrial fed one step at a time over a
socket in the stream’s directory under jobs_dir/streams/), asks it for each step in order (a step whose turn has
not come waits for the earlier ones, running in other rollouts of the batch), and stops it after the last step, an
aborted step, or stream.idle_s without a request. With a frozen adapter, harbor run on the same task evaluates
the whole stream in one trial. Put the stream’s rows in file order with data.shuffle: false. A stream’s driver
outlives the rollout that started it; it runs on the host with the pipeline.
Packaging a benchmark. python -m hermit.harbor.dataset download terminal-bench@2.0 --output <dir> fetches a
registry dataset; a benchmark of your own is written as task directories by your utils/prepare.py (the judge goes
into tests/, uploaded after the agent ran, so the answer is never where the agent can read it; see
examples/officeqa/utils/harbor/build_tasks.py). dataset lock records the sha256 of every task directory and
dataset check compares; dataset splits writes deterministic train/val/test lists by task. The task set is
checked without a model by running it with the scripted agent (acting_agent: {import_path: harborside.scripted:ScriptedAgent, kwargs: {script: "echo 42 > /app/answer.txt"}}): a script that produces the
right answer must score 1 and a wrong one 0 (tests/core/test_harbor_smoke.py does this against Docker).
Limits, each a piece still to build rather than a boundary of the protocol: an adapter whose context comes from a
context_tool is refused, as the CLI agents refuse it (its per-task context would never reach the agent in the
container); the adapter’s tools are not served in the container (no Hermit there; the config’s mcp_servers are
dropped with a log line); what an agent writes into the adapter’s copy in the container (a memory file) does not
come back; a benchmark whose acting model only answers (CL-Bench’s host, OfficeQA’s direct harness) needs a
host-side acting agent whose tools run in the task’s environment, which does not exist yet, so those examples
still run their own loops; hermit eval on a task set still needs the training-only config keys.
What the task owner provides#
Input |
Purpose |
Used by |
|---|---|---|
|
Task dictionaries with unique IDs. |
Pipeline and scoring function. |
Split files |
One task ID per line for train, optional validation, and test. |
Trainer and evaluation. |
|
Description of the task, objective, feedback, and execution. |
Evolver. |
Pipeline callable |
Adapter integration and acting execution. |
Training, evaluation, and candidate experiments. |
Scoring callable |
Score and feedback for a result. |
The same execution paths. |
Optional resources |
Environment paths or task tools granted to the learner. |
Evolver. |
Run configuration |
Connect these inputs with a template and Evolver. |
Framework commands. |
A task dictionary needs a unique id (task_id is also accepted); its other fields belong to your integration.
Split IDs must refer to existing tasks. check detects overlapping splits. A missing test split permits training,
but eval needs test tasks.
task-info#
Provide a README.md as the learner’s entry point. Explain what an input means, what a correct result looks like,
the optimization objective, and how to interpret feedback. Larger tasks can split this into task.md, goal.md,
judge.md, and pipeline.md; these are organizational conventions.
Written task rules inform the learner. They are not automatically enforced by the Trainer’s specification check. Requirements that must be enforced need an appropriate task judge or template check.
Interaction with other components#
The Trainer calls the pipeline on each training task, passes its result to the scoring function, and writes the combined record as learning material. The same functions serve test evaluation and candidate runs requested by the Evolver. They should not rely on one particular caller or call order.
The adapter reaches the acting system through its exports. A context-only integration
may read text into its prompt. A tool or service adapter needs the corresponding host integration and resources.
check does not prove that a host supports every export.
The task environment and learner access are separate. A pipeline may use resources that the Evolver cannot inspect.
evolver_access.environment grants the learner additional paths; evolver_access.tools declares capabilities.
Neither setting configures the acting pipeline for you.
These choices keep task execution reusable when the template or learning method changes, and keep the pipeline’s
original execution available as the adapter=None baseline.
Minimal example#
This complete offline stub loads adapter context and echoes the task input. Its messages demonstrate the exchange; it does not call a model. Replace the echo with your acting system when integrating a real task.
"""Offline pipeline stub for exercising the interface; no model is called."""
from pathlib import Path
from hermit import Session
from hermit.task import RunOutput
def run(task: dict, adapter: Path | None) -> RunOutput:
context = Session(adapter).context() if adapter is not None else ""
# Replace this echo with the acting system that consumes context and the task.
answer = task["prompt"]
return RunOutput(
output=answer,
messages=[
{"role": "system", "content": context},
{"role": "user", "content": task["prompt"]},
{"role": "assistant", "content": answer},
],
cost_usd=0.0,
tokens=0,
)
Its scoring function compares the output with the expected string:
"""Scores the offline echo task."""
from hermit.task import JudgeResult, RunOutput
def score(task: dict, output: RunOutput) -> JudgeResult:
correct = output.output == task["expected"]
return JudgeResult(score=float(correct), feedback="match" if correct else "mismatch")
The example config binds these functions as:
pipeline:
name: pipeline.py:run
judge:
path: judge.py
name: score
Paths are relative to the config file. agent.name includes the class or function name; judge.path names the file
or module and judge.name names the function (default score). agent.params is the agent’s to read.
Run the offline example to exercise these interfaces. For a real model and document environment, see OfficeQA.