Model backends#

Every model a run calls is named by a reference, <backend>/<model id>, and a backend is one endpoint: its base URL, the environment variable holding its key, the API dialects it serves and, optionally, prices per model. A config names models by reference; which endpoint answers is decided in one place, so the same run can be pointed at Fireworks, at OpenAI or DeepSeek, or at a vLLM or SGLang server on your own machine without editing the config, the launchers or the task agents.

model: fireworks/accounts/fireworks/models/glm-5p3-flash   # the run's model: the `llm` tool, the agent and the Evolver
                                                            # use it when they name none of their own
hermit backends list                 # the backends known here
hermit backends show vllm/Qwen/Qwen3-30B-A3B-Instruct-2507-FP8        # a reference, resolved
hermit backends check vllm/Qwen/Qwen3-30B-A3B-Instruct-2507-FP8       # does the server answer each dialect?
hermit train --config run.yaml model=vllm/Qwen/Qwen3-30B-A3B-Instruct-2507-FP8   # the same run, locally

Where a model is named#

Setting

Who calls the model

Dialect it speaks

llm (or evolver.model with evolver.base: direct)

the Evolver’s direct agent and the run’s llm tool: the gepa, skillopt, ace, reharness and continual_harness Evolvers

chat completions

agent.params.model (or agent.model) with agent.name: react

the built-in tool-calling agent

chat completions

agent.params.model (or agent.model) with agent.name: codex / claude-code

the CLI acting agent

Responses API / Messages API

agent.params.model (or agent.model) with agent.name: mini-swe-agent

mini-swe-agent, through litellm

chat completions

evolver.params.model of the codex, claude and mwm Evolvers

the learner’s CLI session

Responses API / Messages API

agent.params.model with agent.name: pi, the model of any method on evolver.base: pi

Pi

chat completions, else the Messages API, else the Responses API: the first the backend serves

agent.params.model with agent.name: opencode, evolver.params.model of the opencode Evolver

OpenCode

chat completions

agent.params.model with agent.name: dsh, evolver.params.model of the dsh Evolver

DeepSeek Harness

chat completions, else the Messages API, else the Responses API: the first the backend serves

agent.params.model with agent.name: hermes, evolver.params.model of the hermes Evolver

Hermes Agent

chat completions, else the Messages API, else the Responses API: the first the backend serves

model (top level)

the default for all of the above when they name none

the examples’ task agents (agent.params.model, or the launchers’ MW_MODEL)

CL-Bench’s litellm host, ServeLearnBench’s loop, the SkillOpt rollout

chat completions

Each accepts a reference string, or a mapping {model: <reference>, price: {...}, extra_body: {...}} when a price or request fields are to be set for this use only. The older mapping {base_url, model, api_key_env, price, extra_body} still works for the chat-completions callers and names the endpoint directly.

A reference’s first segment is the backend; the rest is the model’s id on it, with whatever slashes the backend uses (fireworks/accounts/fireworks/models/glm-5p3-flash, vllm/Qwen/Qwen3-30B-A3B-Instruct-2507-FP8). A backend may define aliases (models: {qwen: Qwen/Qwen3-30B-A3B-Instruct-2507-FP8}), so vllm/qwen works too. For Pi a reference becomes its models.json provider hermit: the backend’s URL, the key as $VARIABLE (Pi reads it at each request, so no file holds the key) and the backend’s price. The cost recorded follows the price (the agent’s own price, else the backend’s), not Pi’s own figure. Pi sends such a model text only; for a model that takes images (a tool’s result, such as a screenshot), set image_input: true in agent.pi or evolver.pi. For OpenCode a reference becomes its config’s provider hermit (OpenCode’s OpenAI-compatible package): the backend’s chat-completions URL and the key as {env:VARIABLE} (so no config holds the key). OpenCode runs a reference only; the cost recorded follows the price, not OpenCode’s own figure.

For DeepSeek Harness a reference becomes a provider hermit of its llm-pi-ai model plugin, in the patch layer the session is started with (an overlay of settings, loaded with --patch): the backend’s URL and the name of the key’s variable (DeepSeek Harness reads the key at each request, so no file holds it). provider in agent.dsh or evolver.dsh replaces the provider a reference makes with your own (api, the dialect as openai-completions, anthropic-messages or openai-responses; baseURL; apiKeyEnv; and models, a list of {id: <the model's id>} that must include the model); a sandbox lets through the reference’s key and host, not yours. DeepSeek Harness reports no cost: the cost recorded follows the price (the agent’s own price, else the backend’s). As with Pi, set image_input: true in agent.dsh or evolver.dsh for a model that takes images. Declare the backend’s context_window for DeepSeek Harness (with your own provider, contextWindow beside the model’s id): without one it assumes 1,000,000 tokens and asks for up to 128,000 output tokens a request, which a smaller vLLM server refuses.

For Hermes Agent a reference becomes the provider hermit of its config.yaml (rewritten for every call): its api_mode is chat_completions, else anthropic_messages (at the backend’s Messages root), else codex_responses, the first the backend serves; the key is named by its variable (key_env, which Hermes reads when it starts, so no file holds the key); the backend’s extra_body goes on the provider, its context_window and max_output_tokens on the model (context_length, max_tokens). Hermes runs a reference only (no login, no provider of its own); provider in agent.hermes or evolver.hermes replaces the one a reference makes with your own (api_mode, base_url, key_env, …). The cost recorded follows the price (the agent’s own price, else the backend’s), not Hermes’s own figure.

A string whose first segment is no backend’s name is not a reference: for a Codex or Claude Code agent it is a model of the CLI’s own provider (gpt-5.4), as before. So a provider id that happens to start with a backend’s name (openai/...) is read as a reference to that backend.

To run on a subscription and no key at all, name the model with a login: Claude Code takes login: claude, Codex login: chatgpt; llm and an acting pipeline’s model take {cli: ..., ...}.

The top-level model is a default: model=vllm/qwen on the command line moves every caller that names no model of its own. The OfficeQA configs set it to the acting agent’s model (GLM 5.3 Flash, which the task sandbox is given the key and host of) and set the learner’s llm (GLM 5.3; the SkillOpt paper used GPT-5.5), so model=vllm/qwen moves the agent and llm=vllm/qwen the learner; a config that sets llm or agent.params.model itself keeps them unless they are overridden too (llm=vllm/qwen agent.params.model=vllm/qwen). An agent Evolver such as codex takes its model from evolver.params.model; evolver.params.model=vllm/qwen moves it.

Where backends are declared#

Three layers, each overriding the one below by name:

  1. the run config’s backends: section, or backends.<name>.<field>=... on the command line;

  2. your own file, ~/.config/hermit/backends.yaml (or the file HA_BACKENDS names): the place for the servers you run, declared once for every example and run;

  3. the framework’s built-ins, hermit/agent/model/backends.yaml: fireworks, openai, anthropic, deepseek, vllm (http://127.0.0.1:8000/v1), sglang (http://127.0.0.1:30000/v1) and ollama.

<NAME>_BASE_URL in the environment moves a backend without a file: VLLM_BASE_URL=http://gpu-box:8000/v1. The key is read from the backend’s api_key_env (<NAME>_API_KEY by default: FIREWORKS_API_KEY, VLLM_API_KEY); a local backend has key_required: false, so an unset variable means no key is sent.

# ~/.config/hermit/backends.yaml
vllm:                                   # the built-in, moved: every other field stays
  base_url: http://gpu-box:8000/v1
  api_key_env: LAB_VLLM_KEY
  models: {qwen: Qwen/Qwen3-30B-A3B-Instruct-2507-FP8}
  prices: {Qwen/Qwen3-30B-A3B-Instruct-2507-FP8: {input: 0, cached_input: 0, output: 0}}   # a bill of zero, not unknown
lab-sglang:                             # a new one, like the built-in sglang at another address
  from: sglang
  base_url: http://gpu-box:30000/v1

Field

Meaning

base_url

the OpenAI-compatible root, normally ending in /v1; chat completions and the Responses API are under it

api_key_env

the environment variable holding the key (default <NAME>_API_KEY)

key_required

false for a server that needs no key (default true)

apis

the dialects served: chat, responses, messages (see below)

messages_url

the root the Anthropic Messages API is under, when it is not base_url without its /v1

litellm

the litellm route prefix, for task agents that call through litellm (fireworks_ai, hosted_vllm)

prices

per model id, {input, cached_input, output} in USD per million tokens; a model without one has an unknown cost (never zero)

extra_body

fields added to every chat request, such as reasoning_effort; Pi sends them as its model’s samplingParams with each chat-completions request, under a session’s own effort; Hermes Agent adds them to its chat-completions requests (its provider’s extra_body; not on the Messages API)

context_window

tokens; Codex is told it as model_context_window, Claude Code as CLAUDE_CODE_MAX_CONTEXT_TOKENS, Pi as contextWindow, DeepSeek Harness as its provider’s contextWindow, Hermes Agent as its model’s context_length

max_output_tokens

tokens one request may answer with; Pi is told it as maxTokens, and otherwise uses its default of 16384: a model that reasons long can be cut before it answers, and that session fails; Hermes Agent is told it as its model’s max_tokens

developer_role

keep (default), or system: a relay in front of the backend renames the developer role Codex uses, for an endpoint that refuses it (the built-in fireworks sets it, as the examples always did; Qwen3.6’s chat template refuses a system message after the first, so a local server keeps the role)

models

aliases: short names for model ids

from

start from a copy of another backend (the examples put a metering relay in front of one this way); the copy keeps the source’s key variable, and a copy at another base_url drops the source’s messages_url

Dialects: what Codex and Claude Code need#

The framework’s own client, the react agent and the examples’ task agents speak chat completions, and so do Pi, DeepSeek Harness and Hermes Agent where the backend serves them, and OpenCode. Codex and Claude Code do not: Codex speaks the OpenAI Responses API only (wire_api = "chat" is rejected since Codex 0.95), and Claude Code speaks the Anthropic Messages API only. A backend’s apis says what it serves, and the framework checks a model’s dialect against its backend before a run starts: hermit check reports it, train refuses to start on a mismatch, and hermit backends check <ref> sends one short request in each dialect (and lists GET /models) so a wrong claim shows up before anything is spent.

Backend

chat

responses (Codex)

messages (Claude Code)

fireworks

✓

✓

✓ (under /inference)

openai

✓

✓

anthropic

✓

deepseek

✓

✓

✓ (under /anthropic)

vllm (0.11.1 or later for /v1/messages)

✓

✓

✓

sglang

✓

✓

✓

ollama (0.14 or later)

✓

✓

✓

A backend that serves chat completions only can still run the framework’s own calls and the benchmarks’ agents, but not a Codex or Claude Code acting agent or Evolver; put a translating proxy (LiteLLM serves /v1/responses and /v1/messages in front of any chat endpoint) between them, declared as its own backend.

Serving the Responses API is not quite enough for Codex. Codex (0.162) sends its sub-agent tools and every MCP server’s tools as namespace tools in each request, and the framework’s learners always have an MCP server. The responses probe of check sends one, so a server that refuses it is reported before a run. What was observed on this framework’s test machine (2026-10):

Server and model

Codex learner

vLLM 0.31, Qwen3 / Qwen3.6 (its standard Responses path flattens namespaces)

works

vLLM 0.31, gpt-oss (its Harmony path)

every request refused: tool type 'namespace' is not supported

SGLang 0.5.22, gpt-oss

accepted, but the tools inside a namespace are dropped (the model cannot call MCP tools) and usage is reported as zero

Pick a model the standard path serves for a Codex learner on a local server. On that path vLLM shows a namespaced tool to the model as <namespace>__<name> (mcp__evolver_tools__check_spec) and maps only that name back; a call by the bare name (check_spec) reaches Codex without its namespace and fails as unsupported call. Qwen3-30B used the flat names. Qwen3.6 used the bare name that the Codex Evolver’s prompt gives (Call the check_spec tool): in a full CL-Bench run its learner sessions all completed, since the Trainer checks every step’s adapter itself, but none of its check_spec calls went through. A model must also be able to carry a long agent session: Qwen3-30B-A3B-Instruct-2507 completed the prompt-only learner’s sessions but looped through a MWM bootstrap session without writing anything.

What the binding gives each CLI:

  • Codex: -c model_provider=<backend> -c model_providers.<backend>.base_url=... .wire_api="responses" and .env_key=<api_key_env> when there is a key (Codex sends no credential otherwise), model_context_window when the backend declares one, and web_search="disabled" off OpenAI. These go in front of evolver.params.args or agent.params.args, so your own -c settings win.

  • Claude Code: ANTHROPIC_BASE_URL (the backend’s Messages root), ANTHROPIC_AUTH_TOKEN (the key as a bearer token, which is what vLLM and SGLang check; a placeholder for a keyless server, since Claude Code asks for a login without one), ANTHROPIC_MODEL and every model alias (ANTHROPIC_DEFAULT_{OPUS,SONNET,HAIKU,FABLE}_MODEL, CLAUDE_CODE_SUBAGENT_MODEL) set to the model, CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 off Anthropic, and CLAUDE_CODE_MAX_CONTEXT_TOKENS when the backend declares a context_window: for a model it does not know, Claude Code assumes 200k tokens and compacts too late for a smaller server (a resumed Evolver session on a 131,072-token vLLM server asked for 99k prompt plus 32k output tokens and failed). Claude Code 2.1.302 took model ids with / in them (Qwen/Qwen3-4B-Instruct-2507 on vLLM, accounts/fireworks/models/... on Fireworks) in print mode; vLLM’s own guide still recommends a plain served name (vllm serve ... --served-model-name qwen3).

A sandboxed session (the learner’s, the acting agent’s) gets its model as Harbor’s agents do: the bound endpoint’s key variable passes into the sandbox, its host joins the sandbox’s allowlist, and an endpoint on the loopback (a meter) is forwarded. A config therefore lists neither the endpoint’s host nor its key variable. What the run keeps is scrubbed of key values. Claude Code’s token is written into the Evolver’s settings as $<api_key_env> and expanded when the agent starts, so the run’s config.yaml records the variable’s name, never the key.

A login reaches a sandbox by itself too: login: chatgpt as a copy of the login, with chatgpt.com and auth.openai.com allowed; login: claude as CLAUDE_CODE_OAUTH_TOKEN, passed by name for the call that runs the session (not as one of the sandbox’s variables), with api.anthropic.com allowed.

A local server#

# vLLM: tool calling needs the model's parser; a plain served name suits Claude Code
vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --port 8000 --tensor-parallel-size 2 \
    --enable-auto-tool-choice --tool-call-parser hermes --served-model-name qwen3 [--api-key ...]
# SGLang: Claude Code's requests carry a `thinking` field, which SGLang's /v1/messages takes only with a reasoning
# parser, and ask for 32k output tokens, so the context length must leave room for them
python -m sglang.launch_server --model-path Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --port 30000 \
    --tool-call-parser qwen25 --reasoning-parser qwen3 --context-length 131072 --served-model-name qwen3
hermit backends check vllm/qwen3                    # /models, then chat, responses and messages
hermit check --config examples/officeqa/gepa/run.yaml model=vllm/qwen3 llm=vllm/qwen3
hermit train --config examples/officeqa/gepa/run.yaml model=vllm/qwen3 llm=vllm/qwen3

The examples’ launchers take the same references: MW_MODEL=vllm/qwen3 VLLM_BASE_URL=http://127.0.0.1:8000/v1 for both the host and the learner, MW_HOST_MODEL / MW_EVOLVER_MODEL for either alone (CL-Bench, ServeLearnBench); --model vllm/qwen3 for the Lean swarm (examples/lean_swarm/launch.sh). Their wire meters sit in front of whichever backend the reference names: python -m hermit.agent.model.backends shell <ref> prints a backend’s origin and path for the meter, and a backends.learner={from: <backend>, base_url: <the meter's URL>} override binds the learner through it.

Prices: a hosted backend’s models have prices in the built-in table; a local model has none, so its cost is recorded as unknown. Give it a price of zero in your backends file when a bill of zero is what you mean. The wire-level token counts are recorded regardless. Claude Code reports a cost of its own, at Anthropic’s prices; for a model on another backend that figure is replaced by the backend’s price (unknown without one), and on a subscription (login: claude) it is not kept: nothing is billed per token.

Context length: a local server is often started with a shorter context than the hosted model has, and a request whose prompt plus max_tokens exceeds it fails (vLLM: HTTP 400, “maximum context length”). The built-in react agent keeps its whole history, so a task whose tool results pile up fails on it: on an OfficeQA setting with search tools over the whole corpus, one question’s history on Qwen3.6-35B-A3B reached 114,689 tokens, and with max_tokens: 16384 the request exceeded a 131,072-token server by one token. Serve the model with its full context (vLLM --max-model-len, SGLang --context-length), or lower agent.params.max_tokens and tool_result_chars.