Model backends#
Every model a run calls is named by a reference, <backend>/<model id>, and a backend is one endpoint:
its base URL, the environment variable holding its key, the API dialects it serves and, optionally, prices per
model. A config names models by reference; which endpoint answers is decided in one place, so the same run can be
pointed at Fireworks, at OpenAI or DeepSeek, or at a vLLM or SGLang server on your own machine without editing the
config, the launchers or the task agents.
model: fireworks/accounts/fireworks/models/glm-5p3-flash # the run's model: the `llm` tool, the agent and the Evolver
# use it when they name none of their own
hermit backends list # the backends known here
hermit backends show vllm/Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 # a reference, resolved
hermit backends check vllm/Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 # does the server answer each dialect?
hermit train --config run.yaml model=vllm/Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 # the same run, locally
Where a model is named#
Setting |
Who calls the model |
Dialect it speaks |
|---|---|---|
|
the Evolver’s |
chat completions |
|
the built-in tool-calling agent |
chat completions |
|
the CLI acting agent |
Responses API / Messages API |
|
mini-swe-agent, through litellm |
chat completions |
|
the learner’s CLI session |
Responses API / Messages API |
|
Pi |
chat completions, else the Messages API, else the Responses API: the first the backend serves |
|
OpenCode |
chat completions |
|
DeepSeek Harness |
chat completions, else the Messages API, else the Responses API: the first the backend serves |
|
Hermes Agent |
chat completions, else the Messages API, else the Responses API: the first the backend serves |
|
the default for all of the above when they name none |
|
the examples’ task agents ( |
CL-Bench’s litellm host, ServeLearnBench’s loop, the SkillOpt rollout |
chat completions |
Each accepts a reference string, or a mapping {model: <reference>, price: {...}, extra_body: {...}} when a price
or request fields are to be set for this use only. The older mapping {base_url, model, api_key_env, price, extra_body} still works for the chat-completions callers and names the endpoint directly.
A reference’s first segment is the backend; the rest is the model’s id on it, with whatever slashes the backend
uses (fireworks/accounts/fireworks/models/glm-5p3-flash, vllm/Qwen/Qwen3-30B-A3B-Instruct-2507-FP8). A
backend may define aliases (models: {qwen: Qwen/Qwen3-30B-A3B-Instruct-2507-FP8}), so vllm/qwen works too.
For Pi a reference becomes its models.json provider hermit: the backend’s URL, the key as $VARIABLE (Pi reads it
at each request, so no file holds the key) and the backend’s price. The cost recorded follows the price (the agent’s
own price, else the backend’s), not Pi’s own figure. Pi sends such a model text only; for a model that takes images
(a tool’s result, such as a screenshot), set image_input: true in agent.pi or evolver.pi.
For OpenCode a reference becomes its config’s provider hermit (OpenCode’s OpenAI-compatible package): the
backend’s chat-completions URL and the key as {env:VARIABLE} (so no config holds the key). OpenCode runs a
reference only; the cost recorded follows the price, not OpenCode’s own figure.
For DeepSeek Harness a reference becomes a provider hermit of its llm-pi-ai model plugin, in the patch layer the
session is started with (an overlay of settings, loaded with --patch): the backend’s URL and the name of the key’s
variable (DeepSeek Harness reads the key at each request, so no file holds it). provider in agent.dsh or
evolver.dsh replaces the provider a reference makes with your own (api, the dialect as openai-completions,
anthropic-messages or openai-responses; baseURL; apiKeyEnv; and models, a list of {id: <the model's id>}
that must include the model); a sandbox lets through the reference’s key and host, not yours. DeepSeek Harness reports
no cost: the cost recorded follows the price (the agent’s own price, else the backend’s). As with Pi, set
image_input: true in agent.dsh or evolver.dsh for a model that takes images. Declare the backend’s
context_window for DeepSeek Harness (with your own provider, contextWindow beside the model’s id): without one
it assumes 1,000,000 tokens and asks for up to 128,000 output tokens a request, which a smaller vLLM server refuses.
For Hermes Agent a reference becomes the provider hermit of its config.yaml (rewritten for every call): its
api_mode is chat_completions, else anthropic_messages (at the backend’s Messages root), else codex_responses,
the first the backend serves; the key is named by its variable (key_env, which Hermes reads when it starts, so no
file holds the key); the backend’s extra_body goes on the provider, its context_window and max_output_tokens on
the model (context_length, max_tokens). Hermes runs a reference only (no login, no provider of its own);
provider in agent.hermes or evolver.hermes replaces the one a reference makes with your own (api_mode,
base_url, key_env, …). The cost recorded follows the price (the agent’s own price, else the backend’s), not
Hermes’s own figure.
A string whose first segment is no backend’s name is not a reference: for a Codex or Claude Code agent it is a
model of the CLI’s own provider (gpt-5.4), as before. So a provider id that happens to start with a backend’s
name (openai/...) is read as a reference to that backend.
To run on a subscription and no key at all, name the model with a login: Claude Code takes
login: claude, Codex login: chatgpt; llm and an acting pipeline’s model take {cli: ..., ...}.
The top-level model is a default: model=vllm/qwen on the command line moves every caller that names no model
of its own. The OfficeQA configs set it to the acting agent’s model (GLM 5.3 Flash, which the task sandbox is given
the key and host of) and set the learner’s llm (GLM 5.3; the SkillOpt paper used GPT-5.5), so model=vllm/qwen
moves the agent and llm=vllm/qwen the learner; a config that sets llm or agent.params.model itself keeps them
unless they are overridden too (llm=vllm/qwen agent.params.model=vllm/qwen). An agent Evolver such as codex takes
its model from evolver.params.model; evolver.params.model=vllm/qwen moves it.
Where backends are declared#
Three layers, each overriding the one below by name:
the run config’s
backends:section, orbackends.<name>.<field>=...on the command line;your own file,
~/.config/hermit/backends.yaml(or the fileHA_BACKENDSnames): the place for the servers you run, declared once for every example and run;the framework’s built-ins,
hermit/agent/model/backends.yaml:fireworks,openai,anthropic,deepseek,vllm(http://127.0.0.1:8000/v1),sglang(http://127.0.0.1:30000/v1) andollama.
<NAME>_BASE_URL in the environment moves a backend without a file: VLLM_BASE_URL=http://gpu-box:8000/v1. The
key is read from the backend’s api_key_env (<NAME>_API_KEY by default: FIREWORKS_API_KEY, VLLM_API_KEY); a
local backend has key_required: false, so an unset variable means no key is sent.
# ~/.config/hermit/backends.yaml
vllm: # the built-in, moved: every other field stays
base_url: http://gpu-box:8000/v1
api_key_env: LAB_VLLM_KEY
models: {qwen: Qwen/Qwen3-30B-A3B-Instruct-2507-FP8}
prices: {Qwen/Qwen3-30B-A3B-Instruct-2507-FP8: {input: 0, cached_input: 0, output: 0}} # a bill of zero, not unknown
lab-sglang: # a new one, like the built-in sglang at another address
from: sglang
base_url: http://gpu-box:30000/v1
Field |
Meaning |
|---|---|
|
the OpenAI-compatible root, normally ending in |
|
the environment variable holding the key (default |
|
|
|
the dialects served: |
|
the root the Anthropic Messages API is under, when it is not |
|
the litellm route prefix, for task agents that call through litellm ( |
|
per model id, |
|
fields added to every chat request, such as |
|
tokens; Codex is told it as |
|
tokens one request may answer with; Pi is told it as |
|
|
|
aliases: short names for model ids |
|
start from a copy of another backend (the examples put a metering relay in front of one this way); the copy keeps the source’s key variable, and a copy at another |
Dialects: what Codex and Claude Code need#
The framework’s own client, the react agent and the examples’ task agents speak chat completions, and so do Pi,
DeepSeek Harness and Hermes Agent where the backend serves them, and OpenCode. Codex and Claude Code do not: Codex
speaks the OpenAI Responses API only (wire_api = "chat" is rejected since Codex 0.95), and Claude Code speaks the
Anthropic Messages API only. A
backend’s apis says what it serves, and the framework checks a model’s dialect against its backend before a run
starts: hermit check reports it, train refuses to start on a mismatch, and hermit backends check <ref> sends
one short request in each dialect (and lists GET /models) so a wrong claim shows up before anything is spent.
Backend |
chat |
responses (Codex) |
messages (Claude Code) |
|---|---|---|---|
|
✓ |
✓ |
✓ (under |
|
✓ |
✓ |
|
|
✓ |
||
|
✓ |
✓ |
✓ (under |
|
✓ |
✓ |
✓ |
|
✓ |
✓ |
✓ |
|
✓ |
✓ |
✓ |
A backend that serves chat completions only can still run the framework’s own calls and the benchmarks’ agents,
but not a Codex or Claude Code acting agent or Evolver; put a translating proxy (LiteLLM serves /v1/responses and
/v1/messages in front of any chat endpoint) between them, declared as its own backend.
Serving the Responses API is not quite enough for Codex. Codex (0.162) sends its sub-agent tools and every MCP
server’s tools as namespace tools in each request, and the framework’s learners always have an MCP server. The
responses probe of check sends one, so a server that refuses it is reported before a run. What was observed
on this framework’s test machine (2026-10):
Server and model |
Codex learner |
|---|---|
vLLM 0.31, Qwen3 / Qwen3.6 (its standard Responses path flattens namespaces) |
works |
vLLM 0.31, gpt-oss (its Harmony path) |
every request refused: |
SGLang 0.5.22, gpt-oss |
accepted, but the tools inside a namespace are dropped (the model cannot call MCP tools) and usage is reported as zero |
Pick a model the standard path serves for a Codex learner on a local server. On that path vLLM shows a namespaced
tool to the model as <namespace>__<name> (mcp__evolver_tools__check_spec) and maps only that name back; a call
by the bare name (check_spec) reaches Codex without its namespace and fails as unsupported call. Qwen3-30B used
the flat names. Qwen3.6 used the bare name that the Codex Evolver’s prompt gives (Call the check_spec tool): in a
full CL-Bench run its learner sessions all completed, since the Trainer checks every step’s adapter itself, but none
of its check_spec calls went through. A model must also be able to carry a long agent session:
Qwen3-30B-A3B-Instruct-2507 completed the prompt-only learner’s sessions but looped through a MWM bootstrap session
without writing anything.
What the binding gives each CLI:
Codex:
-c model_provider=<backend> -c model_providers.<backend>.base_url=... .wire_api="responses"and.env_key=<api_key_env>when there is a key (Codex sends no credential otherwise),model_context_windowwhen the backend declares one, andweb_search="disabled"off OpenAI. These go in front ofevolver.params.argsoragent.params.args, so your own-csettings win.Claude Code:
ANTHROPIC_BASE_URL(the backend’s Messages root),ANTHROPIC_AUTH_TOKEN(the key as a bearer token, which is what vLLM and SGLang check; a placeholder for a keyless server, since Claude Code asks for a login without one),ANTHROPIC_MODELand every model alias (ANTHROPIC_DEFAULT_{OPUS,SONNET,HAIKU,FABLE}_MODEL,CLAUDE_CODE_SUBAGENT_MODEL) set to the model,CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1off Anthropic, andCLAUDE_CODE_MAX_CONTEXT_TOKENSwhen the backend declares acontext_window: for a model it does not know, Claude Code assumes 200k tokens and compacts too late for a smaller server (a resumed Evolver session on a 131,072-token vLLM server asked for 99k prompt plus 32k output tokens and failed). Claude Code 2.1.302 took model ids with/in them (Qwen/Qwen3-4B-Instruct-2507on vLLM,accounts/fireworks/models/...on Fireworks) in print mode; vLLM’s own guide still recommends a plain served name (vllm serve ... --served-model-name qwen3).
A sandboxed session (the learner’s, the acting agent’s) gets its model as Harbor’s agents do: the bound endpoint’s
key variable passes into the sandbox, its host joins the sandbox’s allowlist, and an endpoint on the loopback (a
meter) is forwarded. A config therefore lists neither the endpoint’s host nor its key variable. What the run keeps is
scrubbed of key values. Claude Code’s token is written into the Evolver’s settings as
$<api_key_env> and expanded when the agent starts, so the run’s config.yaml records the variable’s name, never
the key.
A login reaches a sandbox by itself too: login: chatgpt as a copy of the login, with chatgpt.com and auth.openai.com
allowed; login: claude as CLAUDE_CODE_OAUTH_TOKEN, passed by name for the call that runs the session (not as one of
the sandbox’s variables), with api.anthropic.com allowed.
A local server#
# vLLM: tool calling needs the model's parser; a plain served name suits Claude Code
vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --port 8000 --tensor-parallel-size 2 \
--enable-auto-tool-choice --tool-call-parser hermes --served-model-name qwen3 [--api-key ...]
# SGLang: Claude Code's requests carry a `thinking` field, which SGLang's /v1/messages takes only with a reasoning
# parser, and ask for 32k output tokens, so the context length must leave room for them
python -m sglang.launch_server --model-path Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --port 30000 \
--tool-call-parser qwen25 --reasoning-parser qwen3 --context-length 131072 --served-model-name qwen3
hermit backends check vllm/qwen3 # /models, then chat, responses and messages
hermit check --config examples/officeqa/gepa/run.yaml model=vllm/qwen3 llm=vllm/qwen3
hermit train --config examples/officeqa/gepa/run.yaml model=vllm/qwen3 llm=vllm/qwen3
The examples’ launchers take the same references: MW_MODEL=vllm/qwen3 VLLM_BASE_URL=http://127.0.0.1:8000/v1
for both the host and the learner, MW_HOST_MODEL / MW_EVOLVER_MODEL for either alone
(CL-Bench, ServeLearnBench); --model vllm/qwen3 for
the Lean swarm (examples/lean_swarm/launch.sh). Their wire meters sit in front of whichever backend the
reference names: python -m hermit.agent.model.backends shell <ref> prints a backend’s origin and path for the meter,
and a backends.learner={from: <backend>, base_url: <the meter's URL>} override binds the learner through it.
Prices: a hosted backend’s models have prices in the built-in table; a local model has none, so its cost is
recorded as unknown. Give it a price of zero in your backends file when a bill of zero is what you mean. The
wire-level token counts are recorded regardless. Claude Code reports a cost of its own, at Anthropic’s prices; for a
model on another backend that figure is replaced by the backend’s price (unknown without one), and on a subscription
(login: claude) it is not kept: nothing is billed per token.
Context length: a local server is often started with a shorter context than the hosted model has, and a request
whose prompt plus max_tokens exceeds it fails (vLLM: HTTP 400, “maximum context length”). The built-in react
agent keeps its whole history, so a task whose tool results pile up fails on it: on an OfficeQA setting with search
tools over the whole corpus, one question’s history on Qwen3.6-35B-A3B reached 114,689 tokens, and with
max_tokens: 16384 the request exceeded a 131,072-token server by one token. Serve the model with its full
context (vLLM --max-model-len, SGLang --context-length), or lower agent.params.max_tokens and
tool_result_chars.