OfficeQA#
This recipe runs OfficeQA Full (246 questions about U.S. Treasury Bulletins) as the
SkillOpt paper does: the paper’s 50 / 24 / 172 train / validation / test split,
each question’s parsed source pages, SkillOpt’s initial skill as the seed, and exact-match scoring. A GLM agent,
SkillOpt’s own rollout, answers with the adapter’s ADAPTER.md as its skill, in direct chat or in SkillOpt’s Codex
harness, and the gepa or skillopt Evolver trains that skill from the scored answers. The code, configs and
trained adapters are in examples/officeqa/
(README). For an API-only
walkthrough without external dataset assets, use First real-model training; for
local training without model access, Quick Start.
Requirements and configuration#
Complete Installation with the skillopt and gepa dependency groups, activate the virtual
environment, and run the commands from the repository root (dependency groups need pip 25.1+):
python -m pip install -e . --group skillopt --group gepa
Also needed:
authorized access to the gated Hugging Face dataset
databricks/officeqa:officeqa_full.csvandtreasury_bulletins_parsed/with itstransformed/andjsons/directories;raw.githubusercontent.com, from whichprepare.pyfetches the split, or a local copy of it (--split-dir);the models’ backend key:
FIREWORKS_API_KEYfor the configs as written;for the Codex harness only: Docker, npm to fetch Codex’s Linux package, and an acting backend that serves the Responses API (Run in the Codex harness).
The models are backend references (Model backends). The acting agent is GLM 5.3 Flash
on Fireworks (fireworks/accounts/fireworks/models/glm-5p3-flash, the configs’ top-level model) and the configs’
learner (llm) GLM 5.3 on Fireworks, both at reasoning effort medium; the paper’s learner is GPT-5.5 at medium.
model=<backend>/<model id> on the command line moves the acting agent; the configs set llm and keep their learner
unless llm=... is given too. Dollar accounting uses the backend’s prices; verify them for your endpoint.
The agent acts in one of the paper’s two harnesses (agent.params.harness), with the question and its parsed source
pages in either:
|
|
|
|---|---|---|
Configs |
|
|
Tools and files |
glob, read and grep on the question’s source files |
a |
Limits |
24 turns, replies of up to 16,384 tokens |
180 s per session, up to 24 sessions |
Prepare the data#
python examples/officeqa/utils/prepare.py --csv /absolute/path/to/officeqa_full.csv \
--docs /absolute/path/to/treasury_bulletins_parsed
It writes into examples/officeqa/utils/:
tasks.jsonl the 246 questions, without answers
splits/ train.txt, val.txt and test.txt (50 / 24 / 172 ids), and smoke_train.txt, smoke_val.txt
and smoke_test.txt: the first four training, two validation and four test questions
data.json absolute paths of the corpus, the answers and the CSV, and drifted_from_split
seed_adapter/ SkillOpt's initial OfficeQA skill as a prompt-only@1.0 adapter, where every config starts
.private/answers.jsonl the answers, which the judge reads
documents a link to the corpus: what the agent's sandbox shows (`environment.read`); `.private/` is not
The split comes from SkillOpt’s repository at a pinned commit unless --split-dir names a local copy of its
data/officeqa_id_split. Questions, answers and source fields come from the CSV, so they follow the dataset
revision you downloaded; drifted_from_split lists the items whose sources differ from the split’s, which is the
paper’s revision. prepare.py prints the split sizes and those ids. data.json holds absolute paths, so rerun
prepare.py after moving the dataset.
Check, train, evaluate#
Inspect examples/officeqa/gepa/smoke.yaml. It selects prompt-only@1.0, starts from ../utils/seed_adapter,
uses the gepa Evolver with GLM 5.3 as its model, batches of two tasks, two validation tasks, one epoch, and a
training budget of USD 5 / three hours. Usage becomes visible at call boundaries, so the budget is not a
provider-side spending cap. The separate check probe and eval calls are outside the training budget.
hermit check --config examples/officeqa/gepa/smoke.yaml
hermit train --config examples/officeqa/gepa/smoke.yaml
hermit eval --config examples/officeqa/gepa/smoke.yaml --adapter final
hermit eval --config examples/officeqa/gepa/smoke.yaml --adapter ../utils/seed_adapter
hermit eval --config examples/officeqa/gepa/smoke.yaml --adapter none
check runs one training task without an adapter to probe the pipeline and judge; use --no-probe to skip that
model call. train executes two steps. On each step it runs the acting agent, scores the answers, lets GEPA reflect
on them and propose a new skill (kept if it does better on the same tasks, then scored on the validation tasks), and
checks and saves the adapter GEPA hands back. eval separately measures a selected version on the four test tasks,
three passes each (eval.repeats): final, the seed skill the run started from, and no skill. A smoke run checks
that the integration works; four test questions cannot establish a performance gain, and no smoke scores are
reported on this page.
The configs run the acting agent in the task sandbox (environment: its code, data.json and the corpus, the
skillopt package; never .private/ or the judge) and the learner in the evolver sandbox, each given its model’s
key variable and provider host. The in-process gepa method calls the run’s model tool directly;
enabling the sandbox does not turn the whole Python process into an isolated environment.
Paths in a config are relative to its file, so the run is under examples/officeqa/gepa/runs/smoke-gepa/
(Outputs and scoring lists what it holds). Read the trained skill and its card, and deploy
it into a separate generated workspace:
cat examples/officeqa/gepa/runs/smoke-gepa/checkout/ADAPTER.md
cat examples/officeqa/gepa/runs/smoke-gepa/checkout/.hermit/CARD.md
hermit deploy --adapter examples/officeqa/gepa/runs/smoke-gepa/checkout \
--workspace .demo/officeqa-codex --host codex
Inspect its AGENTS.md before starting the host. This checks host integration; effectiveness in another host needs
its own evaluation.
Rerunning train resumes an unfinished run at its last completed step. For another experiment after this one
finishes, choose another name, for example trainer.run_name=smoke-02, and use the same override on the
corresponding eval commands. The framework does not choose the best checkpoint: final is the Evolver’s final
selection, which without one is the last adapter. Both methods here make one: GEPA its best validation candidate,
SkillOpt its best skill, handed back as is (promote_final: false).
Full runs#
Each method folder has six configs, each with its own run name and adapter tree, so runs can train side by side;
do not mix their run-name or tree settings. All use the prompt-only@1.0 template and start from
../utils/seed_adapter; all but the Flash-row and GPT-5.6-Luna configs (run_codex_flash.yaml,
run_codex_luna.yaml: one pass) evaluate three passes per test question. The full runs run eight rollouts at a
time, the smoke runs four.
Config |
Harness |
Train / val / test |
|
Epochs |
Budget |
|
|
|---|---|---|---|---|---|---|---|
|
direct |
4 / 2 / 4 |
2 |
1 |
USD 5, 3 h |
60 |
|
|
Codex |
4 / 2 / 4 |
2 |
1 |
none |
60 |
|
|
direct |
50 / 24 / 172 |
3 |
4 |
USD 40, 30 h |
60 |
|
|
Codex |
50 / 24 / 172 |
3 |
4 |
none |
180 |
|
|
direct |
4 / 2 / 4 |
2 |
2 |
USD 5, 3 h |
90 |
|
|
Codex |
4 / 2 / 4 |
2 |
2 |
none |
90 |
|
|
direct |
50 / 24 / 172 |
40 |
4 |
USD 40, 30 h |
180 |
|
|
Codex |
50 / 24 / 172 |
40 |
4 |
none |
360 |
|
|
Codex 0.162 |
50 / 24 / 172 |
3 |
4 |
none |
180 |
|
|
Codex 0.162 |
50 / 24 / 172 |
40 |
4 |
none |
360 |
|
|
Codex 0.164, ChatGPT login |
50 / 24 / 172 |
3 |
4 |
none |
180 |
|
|
Codex 0.164, ChatGPT login |
50 / 24 / 172 |
40 |
4 |
none |
360 |
|
call_timeout_min is the time, in minutes, the Trainer allows each Evolver call (setup, a step, finalize); Codex
sessions are slower, so run_codex.yaml allows more. gepa/run.yaml uses GEPA’s default reflection minibatch of 3
(data.step_size), Pareto selection over the 24 validation questions and 4 epochs over the 50 training questions
(68 steps), since the paper does not publish its GEPA settings. skillopt/run.yaml uses the paper’s SkillOpt
hyperparameters: 4 epochs of batches of 40 (2 steps each), minibatch 8, merge 8, learning rate 4 decaying to 2,
validation gate, gated slow update, and meta skill, with prompt_env: officeqa and trajectory_view: paper.
The two run_codex_flash.yaml configs are run_codex.yaml with GLM 5.3 Flash as the learner too, one evaluation
pass, and the Codex 0.162.0 image (officeqa-codex:0.162.0): the settings of
the Flash row, shared with the Harness² stream. The two run_codex_luna.yaml configs are
run_codex_flash.yaml with the models changed (gpt-5.6-luna acting on a ChatGPT login, login: chatgpt, and
learning as one Codex session per chat call, llm: {cli: codex, ...} with llm.timeout 3,600 s) and the Codex
0.164.0 image: the Flash row with GPT-5.6-Luna; Continual Harness’s and
ReHarness’s stream_codex_luna.yaml likewise.
Harness² has no training run: it processes the test questions as one stream, with Codex as
the acting agent; see harness2_stream/ for its protocol and result.
skillopt/smoke.yaml runs 4 training questions in 2 epochs of 2 steps. Run it first, then the full configs:
hermit train --config examples/officeqa/skillopt/smoke.yaml
hermit train --config examples/officeqa/gepa/run.yaml
hermit train --config examples/officeqa/skillopt/run.yaml
hermit eval --config examples/officeqa/gepa/run.yaml --adapter final
hermit eval --config examples/officeqa/skillopt/run.yaml --adapter final
hermit eval --config examples/officeqa/skillopt/run.yaml --adapter none
hermit eval --config examples/officeqa/skillopt/run.yaml --adapter ../utils/seed_adapter
Each row of Results is the mean_score of one evaluation, in the folder named after its selector:
Row |
Config |
|
|
|---|---|---|---|
no skill |
|
|
|
initial skill |
|
|
|
GEPA |
|
|
|
SkillOpt |
|
|
|
ReHarness, direct chat |
|
|
|
The Codex column uses run_codex.yaml in the same commands (run names codex-gepa and codex-skillopt), after the
setup below.
Run in the Codex harness#
The SkillOpt paper also runs its OfficeQA agent in a Codex harness. Here every Codex run, of every method, is the
framework’s Codex agent (agent.name: codex, A coding-agent CLI as the acting agent)
with OfficeQA’s task functions (utils/task.py, utils/prompt.md): each rollout is one codex exec session in a
workspace-write sandbox without network or web search, in a fresh workspace with the adapter deployed as hermit deploy deploys it (its context in AGENTS.md, its files under hermit/<id>/, named there; up to 64 KiB of
AGENTS.md, project_doc_max_bytes), the question with its parsed source pages as task.md, and the whole corpus
under docs/; one session of up to 3,600 s, whose last <answer>...</answer> is the answer. Each
method folder has smoke_codex.yaml and run_codex.yaml, its smoke.yaml and run.yaml with that agent and no
budget (and, in run_codex.yaml, a longer call_timeout_min). Codex reaches the acting model through its
backend’s Responses API (or a ChatGPT login, login: chatgpt), with a home of its own and none of your Codex
configuration or instructions.
The Codex-harness numbers published below for no skill, the initial skill, GEPA and SkillOpt (the Codex column of
Results and those rows of the Flash row) were made with SkillOpt’s own Codex harness,
which this agent replaced: the skill as .agents/skills/skillopt-target/SKILL.md with a prompt line asking Codex to
use it, 180 s sessions, a session without an <answer> run again with its reply up to 24 times, and SkillOpt’s own
trace summary for its analyst. Harness², Continual Harness, ReHarness’s Codex stream and Meta-Harness Adapter were
made with this agent’s deployment and session (formerly utils/agent/codex_agent.py, now removed). Compare rows made with one
harness; SkillOpt’s Evolver now builds its Codex trace summary from the session’s event stream.
Each Codex session runs in a Docker container of its own (environment.image, hermit/agent/base/container.py):
Files: only its workspace, a fresh Codex home and the corpus (read-only), with a private
/tmp.Network: the containers sit on an internal Docker network with no route out; their only way out is an HTTPS proxy (
docker/codex/egress_proxy.py) that opens connections to the model provider’s API, refuses the rest, and logs each decision.Codex’s sandbox: Codex’s own workspace-write sandbox runs inside the container. The session’s commands write only the workspace,
/tmpand parts of the fresh Codex home, have no network at all, and run in user, mount and network namespaces of their own. Docker allows a container those namespaces only withSYS_ADMINand without its seccomp and AppArmor profiles, so the container runs that way.Environment: the session’s shell gets only Codex’s core environment variables (no API key), with Codex’s shell snapshot off (it would bring back the whole environment).
Resources: each container is limited to 6 GB of memory, 2,048 processes and a 4 GB
/tmp.
Build the image (Ubuntu 24.04 with python3 and Codex 0.134.0 for Linux arm64; on an x64 host pack
@openai/codex@0.134.0-linux-x64, unpack its openai-codex-0.134.0-linux-x64.tgz, and build with
--build-arg TRIPLE=x86_64-unknown-linux-musl), create the network and start the proxy once:
cd docker/codex
npm pack @openai/codex@0.134.0-linux-arm64 && mkdir -p codex
tar -xzf openai-codex-0.134.0-linux-arm64.tgz --strip-components 1 -C codex
docker build -t officeqa-codex:0.134.0 .
docker network create --internal oq-sandbox
docker run -d --name oq-egress --restart unless-stopped --read-only --network bridge officeqa-codex:0.134.0 \
python3 -u /opt/egress_proxy.py 3128 api.fireworks.ai:443
docker network connect oq-sandbox oq-egress
cd - # back to the repository root
Each config names its image in environment.image and its network and proxy in environment.container
(run_codex.yaml and smoke_codex.yaml officeqa-codex:0.134.0, the Flash row’s configs officeqa-codex:0.162.0;
build that one the same way); override them on the command line (environment.image=...,
environment.container.proxy=...). The proxy above lets through only Fireworks: for
another backend, start one with that backend’s host:port (a plain-HTTP or local endpoint cannot be reached from the
network); for a ChatGPT login, with chatgpt.com:443 auth.openai.com:443, and the model named by the login
(agent.model.ref=null agent.model.name=gpt-6-luna agent.model.login=chatgpt).
docker logs oq-egress lists every connection the sessions’ Codex made or was refused. The agent checks the image
and the network before its first session, the task’s prepare the corpus, and they stop if one is missing.
Eight rollouts at a time per run start eight containers; give Docker’s VM enough CPUs and memory for every run you start
side by side (the Codex-harness runs behind Results were four side by side on 12 CPUs and 24 GB). The
container is root with SYS_ADMIN, so Codex’s sandbox is what keeps a session’s commands in: on macOS, Docker’s VM
(colima, Docker Desktop) shares your home folder by default, so share only the corpus and the Codex workspaces’ folder,
and keep the answers out of both.
hermit train --config examples/officeqa/skillopt/smoke_codex.yaml
hermit train --config examples/officeqa/gepa/run_codex.yaml
hermit train --config examples/officeqa/skillopt/run_codex.yaml
hermit eval --config examples/officeqa/gepa/run_codex.yaml --adapter final
hermit eval --config examples/officeqa/skillopt/run_codex.yaml --adapter final
hermit eval --config examples/officeqa/skillopt/run_codex.yaml --adapter none
hermit eval --config examples/officeqa/skillopt/run_codex.yaml --adapter ../utils/seed_adapter
Codex’s workspaces are under ~/.cache/hermit/officeqa-codex-actor/ (agent.params.workspaces), outside the
checkout, so no repository instructions reach the session; each rollout’s events, commands included, are in its
log/agent-codex.jsonl, which the learning material copies.
Through Harbor#
The same Codex harness runs through Harbor instead of a container of the
acting agent’s own (environment.image): each question is a Harbor task, and Harbor starts the container, installs Codex, deploys the
skill and runs the judge inside, after the session. Build the task set once (private: it holds the answers), then
run a method’s run_codex_harbor.yaml with the corpus mounted where the tasks expect it:
python -m pip install --group skillopt # the task text is rendered by SkillOpt's own code
python examples/officeqa/utils/harbor/build_tasks.py # utils/.private/harbor_tasks/<UID>/, tasks.jsonl, MOUNTS.json
export OFFICEQA_DOCS_ROOT=... # the mount the build prints: the corpus at /app/docs/root_1
hermit check --config examples/officeqa/gepa/run_codex_harbor.yaml
hermit train --config examples/officeqa/gepa/run_codex_harbor.yaml
hermit eval --config examples/officeqa/gepa/run_codex_harbor.yaml --adapter final
What differs from the Codex route above: the container is Harbor’s (ubuntu:24.04 with python3 and ripgrep;
build_tasks.py --base-image takes one with Codex installed, which saves Harbor’s install per trial), its network
is the task’s (no egress proxy: the model’s endpoint and key reach Codex through Harbor), the question and its
parsed pages are the instruction rather than a task.md, and the judge
(utils/harbor/judge.py, SkillOpt’s exact match) reads the <answer> from the session’s final message, or
/app/answer.txt, inside the container. The scores of this route are not in the results table: a run through it is
a different setting and is reported as one.
What the isolation stopped#
GLM sessions look for data the question lacks (exchange rates, price indexes) outside their workspace and on the internet. Of the 3,438 rollouts of the Codex-harness runs behind Results (25,151 commands in their transcripts; a session that times out leaves none), about 200 tried the network (BLS, World Bank, FRASER) and about 310 looked outside their workspace. None got anything: names did not resolve, the image has no curl or wget, and the container holds nothing else. The proxy let through only connections to the model provider and refused Codex’s own to chatgpt.com, ab.chatgpt.com and api.github.com.
A model on a ChatGPT login#
For a model the ChatGPT login serves (the Flash row with GPT-5.6-Luna) the image
carries the machine’s own standalone Codex and the egress proxy lets through chatgpt.com and auth.openai.com. codex login on the machine once: the Codex agent (login: chatgpt) gives each session a copy of the login from one shared
copy (~/.cache/hermit/codex-login/), to which each session’s refreshed token is written back, so a rotated token
reaches the next session. The learners’ calls (llm: {cli: codex, ...}) are each one Codex session on the same
login in a bubblewrap sandbox of its own, started by the Trainer (its empty workspace, the login’s hosts through a
proxy of its own; isolation: container gives it this image and proxy instead); Harness²’s editors are Codex sessions in the run’s sandbox
(evolver.params.backend: codex). Sessions on the login share its plan, and files under
~/.cache/hermit/codex-login/plan/ steer every running session: PAUSE holds new sessions back, a number in
MAX_USED holds them while the plan’s fullest window is at or above that percent, a number in SLOTS caps how many
run at once on the machine; the plan’s latest report is plan.json there. A session the plan refuses is run again
when its window resets, never scored; one that met “model is at capacity” is run again later.
cd docker/codex
cp -a ~/.codex/packages/standalone/releases/0.164.0-x86_64-unknown-linux-musl codex-standalone
docker build -f Dockerfile.standalone -t officeqa-codex:0.164.0 .
docker run -d --name oq-egress-login --restart unless-stopped --read-only --network bridge officeqa-codex:0.164.0 \
python3 -u /opt/egress_proxy.py 3128 chatgpt.com:443 auth.openai.com:443
docker network connect oq-sandbox oq-egress-login
cd -
Claude Code in the same containers#
Claude Code runs in this setup too (the Flash row with Claude Haiku 5.5): a
container per acting session on oq-sandbox (environment.image), an egress proxy of its own that lets through
api.anthropic.com and nothing else, and Claude Code’s own sandbox inside for the session’s commands (bubblewrap, no
network, writes to the workspace and /tmp only), as Codex’s workspace-write sandbox runs in its. The image carries
the machine’s own Claude Code, one file of the version the image is named for; the login’s token reaches Claude Code
on a file descriptor, and the plan stays outside (the guide).
The learners’ calls (llm: {cli: claude, ...}) are each one Claude Code session on the same login in a bubblewrap
sandbox of its own, started by the Trainer (isolation: container gives it this image and proxy instead); Harness²’s
editors are Claude Code sessions in the evolver sandbox (evolver.params.backend: claude):
cp ~/.local/share/claude/versions/2.1.318 docker/claude/claude # the machine's Claude Code, the native build
docker build --build-context codex=docker/codex -t officeqa-claude:2.1.318 docker/claude
docker run -d --name oq-egress-claude --restart unless-stopped --read-only --network bridge officeqa-claude:2.1.318 \
python3 -u /opt/egress_proxy.py 3128 api.anthropic.com:443
docker network connect oq-sandbox oq-egress-claude
ReHarness#
ReHarness (Xiao et al., 2026) learns a bank of reflections, one per question
seen, instead of one skill: before each question the harness retrieves the five reflections whose source questions
are closest to it (Qwen3-Embedding-8B, cosine) and appends them to the agent’s context below ADAPTER.md; after
it, the Evolver revises each retrieved reflection from its own source trajectory with the new one as feedback, and
writes the question’s reflection. The method is strictly online: one pass over a stream, one question per step, no
validation set, no replay (ReHarness has its controls). examples/officeqa/reharness/
runs it in direct chat with the reharness Evolver on the reharness template, starting from SkillOpt’s initial
skill as ADAPTER.md (make_seed.py writes seed_adapter/: the skill, then the template’s framing of the retrieved
reflections, over an empty bank), with GLM 5.3 Flash at medium effort as the acting model and as the reflection and
revision agents, as the paper uses one backbone for every agent. utils/agent/agent.py hands the question to the
adapter’s context tool (Session.context(query=question, task=row)) and records what was retrieved in the output’s
extra["context_tool"].
Two protocols, because the paper’s is not this page’s:
Config |
Stream ( |
Test |
|
The number |
|---|---|---|---|---|
|
the 4 smoke training questions |
the 4 smoke test questions, one pass |
|
none: a smoke |
|
the 50 training questions |
172, three passes |
|
|
|
the 172 test questions |
none |
|
the mean of |
run.yaml is the setting of Results: the bank is written from the 50 training questions in the split’s
order, then frozen and evaluated on the 172 test questions like the GEPA and SkillOpt skills, so its row is
comparable with theirs. stream.yaml is the paper’s protocol: the 172 test questions are themselves the stream;
each is answered with the bank written from the questions before it and scored before the Evolver reads it, so the
judge’s feedback, which names the reference answer, reaches the bank only after the question was answered, as the
environment’s reward does. Its number is not comparable with the Results table (the method learns from the test
questions themselves, and each is answered once), only with other methods run under the same protocol. Neither
protocol uses a validation split: there is no candidate selection, and final is the bank after the stream. All
three configs set data.step_size: 1, data.shuffle: false and trainer.epochs: 1, a budget of USD 5 / 3 h
(smoke) or USD 40 / 30 h, and call_timeout_min: 30.
The adapter’s reader runs in the rollouts’ adapter sandbox, which has no keys and reaches only the loopback
endpoints named in the environment, so the configs are started through launch.sh: it starts a wire meter
(python -m hermit.runtime.wire_meter) on WM_PORT in front of Fireworks that signs the reader’s requests with
FIREWORKS_API_KEY (or reuses one already there), exports REHARNESS_EMBED_URL to it, writes seed_adapter/, and
runs the command with the Evolver’s embedding model bound to the same relay (evolver.params.embedding), so the
meter’s ledger, reharness/runs/wire_ledger-<port>.json, holds the whole embedding bill under the bucket
officeqa_reharness_<config>. The key comes from the environment or the file MW_KEY_FILE names and is written
nowhere. Further arguments go to hermit as they are:
FIREWORKS_API_KEY=... WM_PORT=3601 examples/officeqa/reharness/launch.sh smoke check --no-probe
FIREWORKS_API_KEY=... WM_PORT=3601 examples/officeqa/reharness/launch.sh smoke
FIREWORKS_API_KEY=... WM_PORT=3601 examples/officeqa/reharness/launch.sh smoke eval
FIREWORKS_API_KEY=... WM_PORT=3601 examples/officeqa/reharness/launch.sh run
FIREWORKS_API_KEY=... WM_PORT=3601 examples/officeqa/reharness/launch.sh run eval # --adapter final
FIREWORKS_API_KEY=... WM_PORT=3601 examples/officeqa/reharness/launch.sh stream
eval takes --adapter final unless another selector is given (--adapter seed_adapter is the initial skill over
an empty bank, which retrieves nothing). An interrupted stream is resumed in place by the same command
(resume_mode: auto). The run’s records are those of Outputs and scoring, plus the bank:
<run>/checkout/reflections/<id>.json, one per question of the stream, each with its revisions, and
<run>/workspace/state/reharness.json, what each step retrieved, revised and wrote.
The Codex-harness configurations are reharness/stream_codex.yaml and reharness/run_codex_flash.yaml; the
stream’s number is in the Flash row.
Meta-Harness Adapter#
Meta-Harness Adapter is the outer loop of Meta-Harness (Lee et al., 2026) with the
search confined to the adapter: the acting agent stays fixed, where Meta-Harness also rewrites the agent’s own loop. A
coding-agent proposer reads every earlier candidate’s files, scores and full trajectories and writes new candidates,
each is evaluated on a fixed search set, and the best there is kept (Meta-Harness
Adapter has its controls). examples/officeqa/meta_harness_adapter/ runs it on
the Flash row’s settings: the 50 training questions are the search set (data.step_size: 50, so one step evaluates
one candidate), the proposer is Codex with GLM 5.3 Flash at medium effort, and the acting agent is
the framework’s Codex agent (ADAPTER.md in the session’s AGENTS.md, every other file of the adapter in its
workspace under hermit/meta-harness-adapter/ and named in AGENTS.md). It starts from SkillOpt’s initial skill (make_seed.py writes seed_adapter/) on the
meta-harness-adapter template, whose files outside .hermit/ are all the proposer’s to write.
Config |
Rounds × candidates |
Epochs |
|
The number |
|---|---|---|---|---|
|
1 × 2 |
3 |
|
none: a demo, no test evaluation |
|
6 × 2 |
13 |
|
|
python examples/officeqa/meta_harness_adapter/make_seed.py
hermit train --config examples/officeqa/meta_harness_adapter/run.yaml
hermit eval --config examples/officeqa/meta_harness_adapter/run.yaml --adapter final
The proposer runs in the run’s sandbox, with read access to the corpus and no network but its model’s endpoint; the
acting sessions are containers of their own, as for the other Codex-harness methods. Each proposal step’s
step_NNNN/output/ holds the round’s prompt, the proposer’s session and its report;
<run>/workspace/state/evolution_summary.jsonl lists every candidate with its hypothesis, base and mean on the
search set. The result is in the Flash row.
ACE#
ACE (Zhang et al., ICLR 2026) grows a playbook of bullets with helpful and
harmful counters, one training question at a time: the acting agent names the bullets it used, a Reflector tags
them and diagnoses the attempt (a wrong answer is run again with the reflection, up to three rounds), and a Curator
adds bullets (ACE has its controls and its differences from the authors’ code).
examples/officeqa/ace/ starts from SkillOpt’s initial skill followed by an empty playbook (make_seed.py writes
seed_adapter/) on the ace template, with data.step_size: 1, two epochs over the 50 training questions and
validation at each epoch’s end; final is the playbook with the best validation score.
Config |
Epochs |
|
The number |
|---|---|---|---|
|
1 |
|
none: a smoke, 4 training questions |
|
2 |
|
|
Both are in the GPT-5.6-Luna setting: the acting sessions on the machine’s ChatGPT
login, the Reflector and Curator one Codex session per call (llm: {cli: codex}). The run below is one training with its four
evaluations, one pass each.
Models:
gpt-5.6-lunaacting and learning, reasoning effort medium, on a ChatGPT login.Harness: the Codex agent, a container per session (
officeqa-codex:0.165.0), four side by side.Data: the 50 / 24 / 172 split of SkillOpt
79124b3,data.shuffle_seed: 42;officeqa_full.csvwith SHA-256b5eb1ce0cb34…, eight items drifted from the split (Reproduction limits).Run:
run_codex_luna.yamlof527b130, runluna-ace.Spending: a subscription has no per-token price. Training 1.45 h: 18.5 M tokens of training sessions, 18.5 M of Reflector, Curator, retry and validation sessions. Each evaluation about 15 min.
Adapter |
accuracy |
− the initial skill, points |
|---|---|---|
none |
0.233 |
−14.5 [−20.9, −8.1] |
the initial skill: SkillOpt’s, with ACE’s empty playbook ( |
0.378 |
|
|
0.640 |
+26.2 [+18.6, +33.7] |
the last playbook ( |
0.593 |
+21.5 [+13.4, +29.7] |
Paired per question, 95% bootstrap intervals; bold where the interval lies above zero.
finalagainst the last playbook: +4.7 [−0.6, +10.5] points, 16 questions better and 8 worse. Validation scored 0.708 after both epochs, so the first is kept.The agent names the bullets it used on one line of its final message: 81 of the 100 training samples named at least one, 142 citations in all.
AHE Adapter#
AHE (Lin et al., 2026) evolves the acting agent’s harness over iterations of k
rollouts per task: an Agent Debugger reports on each task’s traces, an Evolve Agent edits the harness and predicts what
each change fixes, and the next iteration’s rollouts judge every change (AHE Adapter has
its controls and its differences from the authors’ code). examples/officeqa/ahe_adapter/ runs it on the ahe
template, with Codex as the acting agent, from the seed harness with SkillOpt’s initial skill in its system prompt (make_seed.py writes
seed_adapter/): the 50 training questions as 100 trial rows (make_rows.py, k = 2), one step per iteration over all
of them, ten iterations; with no validation split, final is the best-ever iteration by training pass rate.
Config |
Steps |
|
The number |
|---|---|---|---|
|
2 |
|
none: a smoke, 4 training questions × 2 |
|
10 |
|
|
Both are in the GPT-5.6-Luna setting: the acting sessions, the Agent Debugger and
the Evolve Agent on the machine’s ChatGPT login, the learners as sessions of the codex Base Agent in the run’s
sandbox. The run below is one training with two evaluations, one pass each.
Models:
gpt-5.6-lunaacting, debugging and evolving, reasoning effort medium, on a ChatGPT login.Harness: the Codex agent, a container per session (
officeqa-codex:0.164.0), eight side by side, 3,600 s each; the adapter’s middleware as Codex hooks.Data: the 50 / 24 / 172 split of SkillOpt
79124b3(validation unused), the training questions in order;officeqa_full.csvwith SHA-256b5eb1ce0cb34…, eight items drifted from the split (Reproduction limits).Run:
run_codex_luna.yamlof725b457, runluna-ahe-adapter.Spending: a subscription has no per-token price. Training 1.5 h: 190 M tokens of rollouts, 122 M of Debugger and Evolve sessions. Evaluations 9 min (
final) and 21 min (H_0).
Adapter |
accuracy |
− H_0, points |
|---|---|---|
H_0: the seed harness with SkillOpt’s initial skill ( |
0.366 |
|
|
0.616 |
+25.0 [+16.9, +33.1] |
Paired per question, 95% bootstrap intervals; bold where the interval lies above zero.
Against the column: GEPA +1.6 [−3.2, +6.4], Harness² +0.6 [−5.2, +6.4], SkillOpt +7.0 [+1.7, +12.4] (GEPA’s and SkillOpt’s four trainings averaged per question first). H_0 against the initial skill alone: +0.6 [−5.2, +6.4].
Answers longer than 40 characters fall from 30% to 0.6%; answers containing the right value anywhere rise from 0.558 to 0.616: most of the gain is the answer’s form.
Training pass@1 by iteration: 0.42, 0.61, 0.65, 0.66, 0.64, 0.64, 0.69, 0.67, 0.63, 0.65; no change was judged HARMFUL, so no automatic rollback ran.
OPRO#
OPRO (Yang et al., ICLR 2024) searches one instruction document, ADAPTER.md on
the prompt-only template, from SkillOpt’s initial skill (utils/seed_adapter). Its optimizer reads the best
instructions scored so far with their training scores and three training problems, and writes four new ones per
step, each scored on the 50 training questions; final is the best validation mean of the top 3 on training
(OPRO).
Config |
Steps |
|
The number |
|---|---|---|---|
|
2 |
|
none: a smoke, 4 training questions |
|
3 |
|
|
Both are in the GPT-5.6-Luna setting: the acting sessions on the machine’s
ChatGPT login, the optimizer one Codex session per call (llm: {cli: codex}). The run below is one training with one
evaluation pass, and one evaluation of the initial skill in the same setting.
Models:
gpt-5.6-lunaacting and learning, reasoning effort medium, on a ChatGPT login.Harness: the Codex agent, a container per session, four at a time, 3,600 s each.
Data: the 50 / 24 / 172 split of SkillOpt
79124b3,data.shuffle_seed: 42.Spending: a subscription has no per-token price. The training took 2.4 h: 27.8 M tokens of training rollouts, 124.5 M of optimizer and candidate sessions. The evaluations took 30 min (
final) and 57 min (the initial skill).
Adapter |
accuracy |
− the initial skill, points |
|---|---|---|
the initial skill ( |
0.355 [0.285, 0.424] |
|
OPRO |
0.436 [0.366, 0.512] |
+8.1 [+1.7, +14.5] |
Paired per question, 95% bootstrap intervals; bold where the interval excludes zero. 24 questions better and 10 worse.
The difference is the answer’s form. Answers that contain the reference value somewhere (an automatic reading that misses some right values: Results) are 118 for the initial skill and 112 for OPRO; long or Markdown answers 55 and 13 of 172. OPRO’s instruction ends “then return only the requested answer format”.
The training means are one pass each; the same instruction scored 0.54, 0.50 and 0.52 at the three steps.
The first evaluation of OPRO’s
finallost 10 of 172 answers to a pipeline error after the session had answered (PermissionErrortidyingdocs/root_1, fixed in the shared layer by a separate pull request); it was discarded and the evaluation run again on the fix, as was the initial skill’s. The training met the same error once; it was run again and scored.
Outputs and scoring#
A run’s records are under examples/officeqa/<method>/runs/<run name>/ (<run> below):
Path |
Contents |
|---|---|
|
the current adapter: |
|
scores, changes, costs and commit IDs per step |
|
results and available logs shown to the learner |
|
the method’s candidates and records, saved per step in |
|
the method’s report for that step |
|
final tag, checkpoints and stop reason |
|
|
|
per attempt: the judge’s score and feedback, |
|
Git history of the adapter versions |
|
direct-chat rollouts: SkillOpt’s prediction files (each call’s own |
|
Codex-harness runs: |
|
every connection the Codex sessions made or were refused |
<selector> is final, none, adapter, or a directory’s name such as seed_adapter; re-evaluating a selector
replaces its records.
Each answer is scored by SkillOpt’s evaluator (
utils/judge.py): exact match after its normalisation (lower case; commas, punctuation and unit words removed), with no numeric tolerance. Token F1 is recorded asf1and is not the score. An attempt without an answer scores 0.Accuracy is
mean_scoreover 172 test questions × 3 passes = 516 attempts per adapter. The framework computes no intervals; those in Results are 95% bootstrap intervals over questions, paired per question.A Codex session longer than 180 s is a wrong answer, as in SkillOpt. SkillOpt’s pinned code reports it as
write() argument must be str, not bytes(its timeout handler writes Codex’s output as bytes), which is also the failure the learners see.A rollout the model API could not serve is run again later; after three attempts it is recorded as an error rather than a wrong answer, counted in the summary’s
errors, and scores 0.The judge’s feedback names the reference answer, and the learners see it for training questions. The validation questions are given to the learners too (each card’s warnings say so), so validation selection is part of training. Test questions are used only by
eval.
Results#
All numbers come from the full configs (run.yaml, run_codex.yaml); none from a smoke run. Agent: GLM 5.3 Flash
(reasoning effort medium) in SkillOpt’s rollout, in direct chat or in the Codex harness (Codex CLI 0.134 for Linux,
one container per session). Learner: GLM 5.3 (medium) for GEPA and SkillOpt; ReHarness’s reflection and revision
agents are the acting model, GLM 5.3 Flash (medium) (ReHarness). Test: 172 questions, three passes each.
Skill |
Direct chat |
Codex harness |
|---|---|---|
none: the agent without an adapter |
0.415 |
0.543 |
SkillOpt’s initial skill ( |
0.465 |
0.564 |
GEPA’s trained skill |
0.564 (card) |
0.595 |
SkillOpt’s trained skill |
0.576 (card) |
0.628 |
ReHarness’s bank ( |
0.579 |
not provided |
Paired differences in points of accuracy, with 95% bootstrap intervals over questions; bold where the interval lies above zero. In direct chat, the trained skills’ wins over the initial skill were reported without their intervals, and initial skill − none was not reported.
Difference |
Direct chat |
Codex harness |
|---|---|---|
initial skill − none |
not reported |
+2.1 [−1.7, +6.0] |
GEPA − none |
+14.9 [+9.5, +20.5] |
+5.2 [−0.2, +10.5] |
SkillOpt − none |
+16.1 [+10.3, +22.1] |
+8.5 [+3.1, +14.0] |
GEPA − initial skill |
+9.9 (interval not listed) |
+3.1 [−1.6, +7.6] |
SkillOpt − initial skill |
+11.0 (interval not listed) |
+6.4 [+1.4, +11.4] |
SkillOpt − GEPA |
+1.2 [−3.1, +5.4] |
+3.3 [−0.8, +7.4] |
The two methods are not separated in either harness. GEPA − none in the Codex harness is borderline: without the items whose sources drifted (Reproduction limits) it becomes +5.8 [+0.6, +11.2]; every other conclusion holds without them.
ReHarness is not in the differences table: the baselines’ per-question results are not in the repository, so no paired interval. Its own 95% bootstrap interval over questions is [0.514, 0.645], which covers both trained skills’ means; its run is described in the example.
The agent starts higher in the Codex harness: it runs shell commands and its own scripts, without direct chat’s turn and reply limits. The initial skill does not help there.
Of each adapter’s 516 Codex-harness test attempts, sessions past 180 s made 12 wrong without a skill, 17 with the initial skill, 6 with GEPA’s and 16 with SkillOpt’s.
For reference, the paper’s Codex harness (GPT-5.5) goes from 38.3 without a skill to 51.1 with SkillOpt; it reports no GEPA in that harness.
What each run handed back as final (validation mean); the direct-chat rows are from the included cards:
Run |
|
|---|---|
GEPA, direct chat |
candidate 1 of 28 (0.708; initial skill 0.500) |
SkillOpt, direct chat |
its best skill, from step 5 (0.750) |
GEPA, Codex harness |
candidate 13 of 18 (0.750; initial skill 0.625) |
SkillOpt, Codex harness |
its one accepted update, from step 1 (0.708); it rejected the rest |
ReHarness, direct chat |
the bank after the 50th question, 49 reflections (strictly online: no validation, no selection) |
Neither Codex-trained skill contains a test answer.
The included cards also record the direct-chat runs’ training time and spending, at the backend’s Fireworks prices:
Run |
Steps |
Training time |
Rollouts |
Evolver |
Test evaluation of |
|---|---|---|---|---|---|
GEPA ( |
68 |
6.48 h |
USD 1.3869 |
USD 7.4196 |
USD 3.4533 |
SkillOpt ( |
8 |
2.40 h |
USD 1.3500 |
USD 4.5337 |
USD 3.4350 |
ReHarness ( |
50 |
1.39 h |
USD 0.3978 |
USD 0.8963 |
USD 3.9676 |
These are observations for planning. Not recorded here:
the cost and time of the no-skill and initial-skill evaluations;
any cost or time of the Codex-harness runs: their skills and cards are not included, and Codex reports only token totals, so the agent records no dollar cost for its rollouts (
cost_usdis null; tokens andwall_sare in eachresults.jsonl);any smoke-run measurement: a smoke budget (USD 5 / three hours) is a cap, not a measurement.
An earlier Codex-harness run on macOS, with Codex’s own sandbox limited to reads of the system, the workspace and
the corpus, gave 0.539, 0.531, 0.591 and 0.597 for the four skills (none, initial, GEPA, SkillOpt). There /tmp
stayed shared and the sessions’ shells inherited the environment of the shell that started the runs; those numbers
are not the results above.
The Flash row#
A second row of settings, on which every method in the repository is measured alike: GLM 5.3 Flash (medium) acts
and learns, every session is Codex 0.162.0 in a container of its own (the image officeqa-codex:0.162.0), the
test questions are evaluated once (eval.repeats: 1, a standard error of about 3.7 points per accuracy), the
learner’s spending is recorded (Codex sessions record only tokens). Not comparable with the table above (Codex
0.134, GLM 5.3 as the learner, three passes). GEPA and SkillOpt were each trained four times from the same config
(gepa/run_codex_flash.yaml, skillopt/run_codex_flash.yaml; run names flash-codex-gepa, -r2, -r3, -r4
and flash-codex-skillopt, -r2, -r3, -r4), each final evaluated once; Meta-Harness Adapter was trained once
(meta_harness_adapter/run.yaml, run mh-flash) and its final evaluated once; Harness² and Continual Harness are
one stream each over the same 172 questions in the same order
(harness2_stream/,
continual_harness/), which answer each question once while learning and are therefore not
evaluations of a frozen adapter.
Method |
Protocol |
accuracy |
Runs |
|---|---|---|---|
none: the agent without an adapter |
one pass |
0.541 |
|
SkillOpt’s initial skill |
one pass |
0.576 |
|
GEPA, |
train on 50 / select on 24 / test on 172 |
0.580 (sd 0.031) |
0.541, 0.610, 0.599, 0.570 |
SkillOpt, |
train on 50 / select on 24 / test on 172 |
0.600 (sd 0.003) |
0.599, 0.605, 0.599, 0.599 |
Meta-Harness Adapter, |
search on 50 / test on 172 |
0.640 |
one training |
Continual Harness, the stream’s answers |
stream over the 172, no training split, no judge verdict |
0.634 |
one stream (the same seed-0 order) |
ReHarness, the stream’s answers |
stream over the 172, no training split, no judge verdict |
0.634 |
one stream (the same seed-0 order) |
Harness², deliverable |
stream over the 172, no training split, no judge verdict |
0.651 |
one stream (seed 0); its reference execution of the initial skill: 0.576 |
Paired per question, 95% bootstrap intervals; a method’s four runs are averaged per question first:
Difference |
points |
|---|---|
initial skill − none |
+3.5 [−2.3, +9.9] |
GEPA − none |
+3.9 [−2.3, +10.5] |
SkillOpt − none |
+6.0 [−0.3, +12.4] |
GEPA − initial skill |
+0.4 [−5.4, +6.2] |
SkillOpt − initial skill |
+2.5 [−3.6, +8.6] |
SkillOpt − GEPA |
+2.0 [−1.0, +5.1] |
Harness² deliverable − its reference execution |
+7.6 [+1.7, +13.4] |
Continual Harness − Harness²’s reference execution |
+5.8 [−1.2, +12.8] |
Continual Harness − Harness² deliverable |
−1.7 [−7.6, +4.1] |
ReHarness stream − Harness²’s reference execution |
+5.8 [−1.2, +13.4] |
Meta-Harness Adapter − Harness²’s reference execution |
+6.4 [+1.2, +11.6] |
Meta-Harness Adapter − Continual Harness |
+0.6 [−5.8, +7.0] |
Meta-Harness Adapter − ReHarness stream |
+0.6 [−5.2, +6.4] |
GEPA’s four trainings spread over 7 points of test accuracy for the same validation score (its
finalwas candidate 3 of 21, 1 of 11, 9 of 23 and 12 of 23, validation means 0.708, 0.708, 0.750, 0.792; the initial skill’s validation mean itself ranged 0.458–0.625 across the runs’ own evaluations of it); SkillOpt’s fourfinalskills (its best skill from steps 1, 6, 2, 1; validation 0.750, 0.792, 0.750, 0.750) are different texts of 3.2–13.9 KB that differ on 27–39 questions each while landing on the same total. Report the means, never one run.The learned skills quote training questions’ answers and particulars (bare-number examples, a month count, a table’s row label; Meta-Harness Adapter’s
finalquotes two as bare-number examples); nofinalskill quotes a test answer (one quoted training answer, 2.916, is also a test question’s).Of each evaluation’s 172 sessions, 1–4 reached the harness’s 180 s cap and scored 0 (4 without a skill); about one session in ten tried the network or searched the host’s file system before answering, and found neither (the container holds the workspace and the read-only corpus; the session’s commands have no network).
Learner spending and training time, at the backend’s prices: GEPA USD 0.09, 0.07, 0.06, 0.08 in 3.2, 2.5, 2.9, 3.1 h; SkillOpt USD 0.15, 0.17, 0.15, 0.14 in 1.4, 1.3, 1.2, 1.3 h; Harness² USD 25.86 of editor sessions and USD 1.62 of executions in 20.5 h, about 0.16 USD per question against GEPA’s and SkillOpt’s 0.01 (the executions’ Codex tokens are not priced); Continual Harness USD 3.67 of Refiner calls and USD 2.81 of executions in 8.6 h, about 0.04 USD per question; Meta-Harness Adapter USD 8.39 of search-set sessions (650: 13 candidates × 50 questions) and USD 0.17 of proposer sessions in 1.5 h, and USD 1.82 for its test evaluation.
Meta-Harness Adapter’s proposer wrote six rounds of two candidates; the starting skill scored 0.60 on the search set, the candidates 0.64–0.72;
finalwas 12 points above the starting skill there (0.72) and 6.4 above it on the test questions, the best of 13 candidates on 50 questions being an optimistic estimate. Every candidate changedADAPTER.mdalone; no script or other file was written.ReHarness’s stream (
stream_codex.yaml,feedback: false, Codex’sAGENTS.mdlimit raised to 200 KiB) scores 0.640 on the first half and 0.628 on the second; 10.0 h, USD 2.43 of sessions and USD 4.81 of reflection and revision calls (about 0.04 USD per question), no errors. With no verdict shown, the reflections still quote numbers: 102 of the 172 contain their question’s reference answer, because they record the agent’s own answers and about two in three of those were right.The two streams are within one stream’s noise of each other (11 questions only Continual Harness gets right, 14 only Harness²); Continual Harness scores 0.663 on the first half of the stream and 0.605 on the second, Harness² 0.640 and 0.663.
Of the misses in each evaluation, about nine are the right number in the wrong form, mostly a missing
%where the reference answer carries one (13 of the 172 do, and the reference answers are not consistent about it); the judge keeps%and strips commas,$andmillion.
The Flash row with GPT-5.6-Luna#
The seven methods of the Flash row once more, with Codex 0.164.0 and GPT-5.6-Luna (reasoning effort medium) acting
and learning on a machine’s own ChatGPT login (a Codex plan) instead of GLM 5.3 Flash on Fireworks. The
example’s results have the table and
what it shows; each method’s run_codex_luna.yaml (stream_codex_luna.yaml for the Continual Harness and ReHarness
streams) is its Flash-row config with the models changed, run with the same commands
(Run; the image, the proxy and the
login: A model on a ChatGPT login). The data, the split, the seed skill, the
judge, the
Evolvers’ parameters, the session limits (SkillOpt’s 180 s × 24 sessions for the trainings and the reference
evaluations, 3,600 s for the streams), the configured concurrency, ReHarness’s embedding model (Qwen3-Embedding-8B on
Fireworks behind its launcher’s wire meter: the plan serves no embeddings) and the one test pass are the Flash row’s;
Meta-Harness Adapter was not run. RRSI Adapter ran later (rrsi_adapter/run_luna.yaml and run_luna_eng.yaml), on
the agent layers: the framework’s Codex agent with its deployment (ADAPTER.md in AGENTS.md, every other file under
hermit/rrsi-adapter/ and named there), the proposer and the search roles sessions of the codex Base Agent, behind
an egress proxy that lets through chatgpt.com only (agent.codex.container.proxy and sandbox.allow on the command
line; today’s keys: environment.container.proxy and sandbox.evolver.allowed_hosts). What differs:
Setting |
The Flash row |
With GPT-5.6-Luna |
|---|---|---|
acting model |
|
|
Codex’s tools and prompt for the model |
GLM is not in Codex’s model catalog: default function tools ( |
Codex fetches gpt-5.6-luna’s catalog entry from chatgpt.com at each session start: code mode only (a JavaScript |
learner ( |
a chat completion to the same GLM (system prompt as such, |
|
Harness² editors |
Claude Code ( |
Codex sessions in hermit’s sandbox ( |
Codex |
0.162.0 (npm package), image |
the machine’s standalone 0.164.0, pinned, image |
egress |
|
|
concurrency on the machine |
each run’s own |
each session on the login, acting or learning, holds one of the machine’s session slots ( |
run names, workspaces |
|
|
The model change cannot be separated from the second, third and fourth rows: they come with running GPT-5.6-Luna in Codex on the plan.
Method |
Protocol |
accuracy [95% bootstrap interval over questions] |
Runs |
|---|---|---|---|
none: the agent without an adapter |
one pass |
0.308 [0.238, 0.378] |
4 sessions at the 180 s cap, scored 0 |
SkillOpt’s initial skill |
one pass |
0.360 [0.291, 0.430] |
3 at the cap |
GEPA, |
train on 50 / select on 24 / test on 172 |
0.600 [0.533, 0.666] (sd 0.017) |
0.593, 0.605, 0.622, 0.581 |
SkillOpt, |
train on 50 / select on 24 / test on 172 |
0.546 [0.481, 0.612] (sd 0.035) |
0.529, 0.523, 0.599, 0.535 |
Continual Harness, the stream’s answers |
stream over the 172, no training split, no judge verdict |
0.477 [0.401, 0.552] |
one stream (the seed-0 order) |
ReHarness, the stream’s answers |
stream over the 172, no training split, no judge verdict |
0.517 [0.442, 0.593] |
one stream (the seed-0 order) |
Harness², deliverable |
stream over the 172, no training split, no judge verdict |
0.611 [0.535, 0.680] |
one stream (seed 0); its reference execution of the initial skill: 0.390 [0.320, 0.465] |
RRSI Adapter, H_T |
evolve on the 50 (two trials each) / test on 172 |
0.616 [0.552, 0.680] (sd 0.029) |
0.634, 0.640, 0.576, 0.616 ( |
Paired per question, 95% bootstrap intervals, a method’s four runs averaged per question first; bold where the interval excludes zero; the Flash row’s GLM intervals beside them:
Difference |
GPT-5.6-Luna |
won / lost |
GLM 5.3 Flash |
|---|---|---|---|
initial skill − none |
+5.2 [−0.6, +11.0] |
18 / 9 |
+3.5 [−2.3, +9.9] |
GEPA − none |
+29.2 [+22.1, +36.5] |
73 / 16 |
+3.9 [−2.3, +10.5] |
SkillOpt − none |
+23.8 [+17.6, +30.2] |
65 / 14 |
+6.0 [−0.3, +12.4] |
GEPA − initial skill |
+24.0 [+16.7, +31.1] |
67 / 17 |
+0.4 [−5.4, +6.2] |
SkillOpt − initial skill |
+18.6 [+12.1, +25.1] |
57 / 19 |
+2.5 [−3.6, +8.6] |
SkillOpt − GEPA |
−5.4 [−9.4, −1.5] |
26 / 44 |
+2.0 [−1.0, +5.1] |
Harness² deliverable − its reference execution |
+22.1 [+15.7, +29.1] |
41 / 3 |
+7.6 [+1.7, +13.4] |
Continual Harness − Harness²’s reference execution |
+8.7 [+1.7, +15.7] |
28 / 13 |
+5.8 [−1.2, +12.8] |
ReHarness − Harness²’s reference execution |
+12.8 [+5.8, +20.3] |
32 / 10 |
+5.8 [−1.2, +13.4] |
Continual Harness − Harness² deliverable |
−13.4 [−20.3, −6.4] |
9 / 32 |
−1.7 [−7.6, +4.1] |
ReHarness − Harness² deliverable |
−9.3 [−15.7, −2.9] |
8 / 24 |
|
ReHarness − Continual Harness |
+4.1 [−2.3, +10.5] |
19 / 12 |
|
RRSI Adapter − its starting adapter (same harness, same window) |
+26.2 [+20.1, +32.6] |
63 / 13 |
|
RRSI Adapter, engineering weights − its starting adapter (same harness, same window) |
+26.7 [+19.8, +33.7] |
49 / 3 |
|
RRSI Adapter − GEPA |
+1.6 [−1.6, +4.8] |
32 / 27 |
|
RRSI Adapter − SkillOpt |
+7.0 [+3.3, +10.6] |
45 / 17 |
|
RRSI Adapter − Harness² deliverable |
+0.6 [−4.2, +5.5] |
23 / 33 |
|
RRSI Adapter − ReHarness |
+9.9 [+4.5, +15.6] |
37 / 26 |
|
RRSI Adapter − Continual Harness |
+14.0 [+8.3, +19.6] |
43 / 20 |
|
RRSI Adapter, engineering weights − RRSI Adapter |
+0.6 [−4.2, +5.5] |
35 / 21 |
|
RRSI Adapter − RRSI Adapter before the port |
+1.5 [−1.9, +4.8] |
36 / 28 |
The differences against no skill and the initial skill are a little high: those two evaluations lost 4 and 3 sessions to the login’s rate limit (at most 2.3 and 1.7 points; by the cells’ own rates about 0.7 and 0.6, by those questions’ scores in the other nine one-pass evaluations about 1.1 and 0.7). Continual Harness’s and ReHarness’s gains over the reference execution overlap GLM’s intervals: they cannot be called larger than GLM’s.
Where the points go. Every answer was checked one by one (2,408: 172 per cell, 688 for GEPA’s and SkillOpt’s four trainings together) for whether its value is right and whether its form is the one the judge accepts; RRSI Adapter’s runs before the port were checked afterwards on their 517 answers that are not exact matches (317 distinct), and its runs on the agent layers on their 440 (284 distinct; the 116 an earlier check had read, the same answer to the same question, kept its verdict); an exact match has the right value in the accepted form:
Cell
exact match
right value, other form
wrong value
no answer (timed out)
right value
no skill
53
62
53
4
0.669
initial skill
62
56
51
3
0.686
GEPA, four trainings
413
53
222
0
0.677
SkillOpt, four trainings
376
77
235
0
0.658
Continual Harness
82
36
54
0
0.686
ReHarness
89
26
57
0
0.669
Harness², deliverable
105
17
50
0
0.709
Harness², reference execution
67
55
50
0
0.709
RRSI Adapter, its starting adapter (same harness)
61
56
55
0
0.680
RRSI Adapter, four trainings
424
66
197
1 (the model’s output limit, not a timeout)
0.712
RRSI Adapter, engineering weights
107
16
49
0
0.715
Paired on the value alone, GEPA − initial skill is −0.9 [−5.7, +3.8], SkillOpt − initial skill −2.8 [−7.7, +2.0], Harness² − its reference execution 0.0 [−4.7, +4.7], Continual Harness − the reference execution −2.3 [−7.6, +2.9] and ReHarness − the reference execution −4.1 [−9.9, +1.7], RRSI Adapter (four trainings) − its starting adapter +3.2 [−0.1, +6.7] (before the port, +0.7 [−3.5, +4.9]; checked the same way, answer by answer, after the column): the learned adapters turn “right value, other form” into exact matches and little else. Two automatic readings (the reference answer at the start of the answer; anywhere in it) never pass a wrong answer but miss many right values (a
%the reference answer has or lacks, 109 answers; list separators 23; a final period 22; trailing zeros 20; a U+2212 minus sign in the reference answer 17), hence the check by hand. Six reference answers contradict the form their question asks for (UID0160, 0225, 0089, 0013, 0196, 0120): about 3.5 points no method can take. GLM writes the bare value by default: a GLM 5.3 Flash evaluation of the same initial skill in the same harness on this machine (runflash-codex-initial-skill, 0.564; not the Flash row’s run) has 19 rejected answers that contain the normalised reference value, against 54 with GPT-5.6-Luna (56 by hand).What was learned. The initial skill says nothing about form (“Copy the final answer from a checked value, not from an unverified intermediate guess”). Afterwards the skill from GEPA’s first training says “Output only the requested answer. Do not include explanations, labels, citations, Markdown, XML/HTML tags, JSON wrappers, …” and the other three carry a line to the same effect (“Return only the requested answer …”); 17 of the 21 harnesses in Harness²’s library carry such a line; Continual Harness’s
AGENTS.mdsays “Put only the normalized answer value and any required unit words inside the tags”, the unit words leaving one answer in ten in a form the judge rejects; ReHarness’sADAPTER.mdis unchanged and its advice is spread over the 172 retrieved reflections, at times contradictory. Bare-value answers: 94–98% of GEPA’s, 79–94% of SkillOpt’s, 83% of Harness²’s, 65% of ReHarness’s, 52% of Continual Harness’s, 37–44% in the three cells without a learned skill.Selection. SkillOpt’s four
finalskills are its best by validation, from steps 2, 7, 4 and 1 (validation means 0.667, 0.708, 0.708, 0.667); GEPA’s from 46, 46, 44 and 44 candidates. GEPA’s four finals spread 0.581–0.622 (sd 0.017), SkillOpt’s 0.523–0.599 (sd 0.035): SkillOpt’s best training (0.599) is above GEPA’s worst (0.581), and over trainings SkillOpt − GEPA is marginal (p ≈ 0.05) where the per-question interval excludes zero.Time and the login. The first wave of trainings (
luna-gepa,-r2,luna-skillopt,-r2) trained 11–15 h while the login was in a slow state (first replies a median 33–54 s, against 1.4–2.1 s afterwards) and learned from 0.8–2.1% timed-out rollouts; the second wave (-r3,-r4) from 0–3 timeouts; the eight evaluations of trained skills ran after the recovery with no timeouts. Two seed-validation rollouts ofluna-gepawere lost to “Selected model is at capacity” and scored as wrong, before the agents learned to run such a session again. With the eight trainings side by side and no limit (about sixty sessions at once) the backend answered 429 Too Many Requests, first replies took a median 100–400 s and 32–45% of three successive batches’ training rollouts timed out; those batches were discarded. The runs behind the results held every SkillOpt-harness rollout and every training learner call to one of a machine-wide number of session slots (hermit/agent/base/login.py, the fileSLOTS), set by a controller (not in the repository) that cut two slots on timeouts and added one after quiet minutes; the slots kept the overload from returning but did not lower the timeout rate while the login was slow (4.4% at 2 slots, 4.6% at 3–5): its recovery did.Tokens, no prices (the plan bills none; a session’s count is Codex’s input, cached included, plus output): 3.4–4.0 M per 172-question evaluation; per training 4.1–4.5 M of training rollouts, 0.6–1.1 M of learner calls (GEPA 0.6–0.7 M, SkillOpt 1.0–1.1 M) and, the bulk of a training, the sessions the Evolver runs to score its candidates on the validation questions and its minibatches: GEPA 20–23 M, SkillOpt 7.2–7.7 M; per stream 46 M of sessions and 11 M of learner calls for Continual Harness, 29 M and 25 M for ReHarness, and for Harness² 27 M of reference executions, 83 M of candidate executions (three per question, the deliverable among them) and 197 M of editor sessions. All runs together took the plan’s weekly window from 29% to 38% (whole percents, account-wide: an upper bound on their share); no session was refused for the plan’s limit and the credits balance did not change.
Two defects of the ports, found on this row and not fixed here, which touch the GLM row too:
utils/task-info/README.mdtells the learner the judge removes “unit words”, wherenormalize_answerremoves million, billion, dollar and nominal and keeps%, thousand and trillion; only Continual Harness’s Refiner reads it, and it wrote “required unit words” into its policy (steps 2–172; “any required unit words” from step 5). The Continual Harness Evolver counts Codex’s own notices (the hook-trust warningcodex_agent.pyalways causes, reconnects) as failure signatures in every step, where upstream sends tool failures to its sub-agent and skill passes only; the Refiner never sees the acting prompt, and from step 84 its policy banned the note writing that prompt asks for (the agent wrote notes in 83 of 83 sessions before, 8 of 89 after).
The Flash row with Claude Haiku 5.5#
The no-skill evaluation once more, with Claude Code 2.1.318 and Claude Haiku 5.5 (effort medium) acting on a Claude
subscription instead of Codex on a ChatGPT login, in the same containers, and the Harness² stream set up to act and
learn the same way. The
example’s results have the column and
what it shows; skillopt/run_claude_haiku.yaml and harness2_stream/run_claude_haiku.yaml are the luna configs with
the agent and the models changed, run with the commands in
Run
(the sign-in, the plan and the boundaries: the guide). The data, the split, the
seed skill, the judge, the prompt, the task functions, the Harness² stream order, H0, its parameters and stage prompts,
the 3,600 s session clock and the one test pass are the luna column’s. The runs below ran before the one sandbox design
(#78), under its keys at the time: the acting agent’s isolation: container with container, today
environment.image and environment.container, and the rollout role’s bubblewrap sandbox, today the task sandbox.
What differs:
Setting |
With GPT-5.6-Luna |
With Claude Haiku 5.5 |
|---|---|---|
acting agent |
Codex 0.164.0 ( |
Claude Code 2.1.318 ( |
the agent’s tools |
Codex’s code mode (a JavaScript |
Claude Code 2.1.318’s default set (Bash, Read, Edit, Write, NotebookEdit, Task for sub-agents, Skill, ToolSearch and the rest it ships; no Glob or Grep in this build: search goes through Bash), WebSearch, WebFetch and the tools whose work outlives a session denied (the no-skill sessions called Bash and Read only); the commands ran in Claude Code’s own sandbox, as Codex’s in its, and had no network. The sessions also carry Claude Code’s 17 built-in skills and 5 built-in agents: “no skill” is no adapter, not no skills |
sign-in and plan |
|
|
no skill |
SkillOpt’s former Codex harness, 180 s sessions re-run up to 24 times without an |
the framework’s agent ( |
Harness² executions |
Codex sessions, read by upstream’s Codex renderer |
Claude Code sessions, each in a container as above, running the same |
Harness² editors |
Codex sessions ( |
Claude Code sessions ( |
learner ( |
|
|
run names |
|
|
No skill. One pass (--adapter none trainer.run_name=haiku-none, 2026-11-05, every session in a container of its
own): 0.599 [0.523, 0.669], 103 of 172, no error. The 172 sessions took 427 s, 8 at a time (13.7 s each at the
median, 166.5 s at the longest, 4 turns at the median); they called Bash 637 times and Read 9 times, no other tool, and
used 24.9 M tokens (20.5 M read from the prompt cache, 3.9 M written to it, 0.5 M of output; 97 K a question at the
median). Claude Code’s sandbox adds a section to each session’s first message (how it is set up, and that a task that
cannot be finished with what was given should say what is missing) and to the Bash tool’s description: a session’s first
request was 18.9 K tokens at the median, against 15.6 K without it. Every session ran on the login
(apiKeySource: none) and on Claude Haiku 5.5 alone (the CLI’s init and modelUsage), none met a limit or drew usage
credits, and no record holds the token. Three earlier passes, on earlier versions of this change, scored 0.640 [0.570,
0.709] in containers, before the review’s last fixes (they change a session’s first request by one number), and 0.622
[0.547, 0.692] and 0.634 [0.558, 0.703] in the rollout role’s bubblewrap sandbox instead of a container, whose sessions
were offered tools a session is now denied (all seven in the 0.634 pass, ListAgents and SendMessage in the 0.622
one). The four average 0.624 (standard deviation 1.8 points). This pass less each, paired: −4.1 [−9.3, +1.2],
−2.3 [−7.6, +2.9] and −3.5 [−8.7, +1.7]; the two passes in containers less the two without, per question: −0.9
[−4.1, +2.3]. 89 questions are right in all four passes and 47 in none. The isolation moved the score no more than
another sample does, and the samples differ in the answer’s form, not in its value (below).
After the one sandbox design. The same evaluation on this code after #78 (the acting agent’s container from
environment.image and environment.container, the learner in a bubblewrap sandbox of its own; 2026-11-07, run
haiku-none): 0.616 [0.547, 0.686], 106 of 172, no error, 413 s at 8 at a time. Every session ran on the login
and on Claude Haiku 5.5 alone, none on usage credits; Bash 655 and Read 9 calls; 26.0 M tokens (21.5 M read from the
prompt cache; 88 K a question at the median); Claude Code’s sandbox refused 13 connections to data hosts, the
egress proxy let 519 through to api.anthropic.com and saw no other host; no record holds the token. Paired with
the four passes above: +1.7 [−3.5, +7.0] against 0.599, −2.3 [−7.6, +2.9] against 0.640, −1.7 [−6.4, +3.5] against
0.634, −0.6 [−5.8, +4.7] against 0.622, and −0.7 [−4.9, +3.5] against their mean per question; the five average
0.622 (standard deviation 1.6 points). Its misses were not checked by hand.
Where the points go. The 69 misses were checked one by one, as the luna column’s answers were (31 are the answer an earlier pass gave to the same question, and kept its verdict):
Cell
exact match
right value, other form
wrong value
no value
right value
Claude Haiku 5.5, no skill
103
20
44
5 (the session said the corpus lacks an input)
0.715
GPT-5.6-Luna, no skill (above)
53
62
53
4 (timed out)
0.669
The 20 are a
%the reference answer has or lacks (7), words or units around the value (6), trailing zeros (3), a percent where the question asks for a decimal (2), a list separator (1) and a value not rounded as asked (1); four of them (UID0160, 0225, 0089, 0120) are among the six reference answers that contradict the form their question asks for. Both passes in containers have the value right in 0.715 of the questions (the 0.640 pass with 110 exact matches and 13 in another form), and the 0.622 pass, checked the same way, in 0.703: the passes differ in how often the value comes in the judge’s form, not in how often it is right. Without a skill Claude Haiku writes the value in the form the judge accepts far more often than GPT-5.6-Luna, and has the value right a little more often (0.715 against 0.669, not paired): its 29 points over luna’s no skill are almost all the form, as the luna column’s gains are.What the isolation refused. The sessions’ commands tried 14 connections, all for data the corpus lacks: FRED (7), BLS (3), the World Bank (2), FRASER and Macrotrends. Claude Code’s sandbox refused each inside the container, so none reached the egress proxy, which let through 517 connections, all Claude Code’s own to api.anthropic.com, and refused none. The five sessions with no value each said which input they could not find or trust (a chart only in an image, the January 2003 Aa yield, a 1968 gilt yield, 1961’s unemployment rates, a 1959 exchange rate) instead of answering without it.
Harness². Not run in full. Its stream (run haiku-h2-s0, on this change before its review, with the executions in
the rollout role’s bubblewrap sandbox: before the containers) was started and stopped during its third question, with
no summary. Its first two questions ran end to end: every stage on Claude Code sessions
on the login, no stage falling back and no editor failing, and in the second the deliverable’s composed skill deployed
in .claude/skills/ and called by its execution through Claude Code’s Skill tool. A two-question smoke before it
(MAX_TASKS=2, run haiku-h2-smoke) ran on a still earlier version, whose sessions had all of Claude Code’s tools. The
column has no Harness² cell.
Included adapters#
examples/officeqa/gepa/adapter/ and examples/officeqa/skillopt/adapter/ are the skills gepa/run.yaml and
skillopt/run.yaml trained in direct chat. examples/officeqa/ace/adapter/ is the final playbook of
ace/run_codex_luna.yaml’s measured run (ACE), its card with the run’s settings and test score. Evaluate them with a
directory selector:
hermit eval --config examples/officeqa/gepa/run.yaml --adapter adapter
hermit eval --config examples/officeqa/skillopt/run.yaml --adapter adapter
hermit eval --config examples/officeqa/ace/run_codex_luna.yaml --adapter adapter
Numbers they copied from training answers are replaced by made-up numbers of the same form, so their cards’ scores are those of the adapters as trained; re-evaluate the distributed adapters when that distinction matters.
No ReHarness bank is included: its reflections quote the training questions’ reference answers from the judge’s
feedback (37 of the 49 in the direct-chat run.yaml bank), which the repository does not redistribute. Each
ReHarness run’s card is in its checkout (reharness/runs/<run>/checkout/CARD.md), with the bank beside it.
Reproduction limits#
This setup differs from the paper’s:
Models: GLM 5.3 Flash acts and GLM 5.3 learns, instead of the paper’s GPT-5.x and Qwen models.
Tools: SkillOpt’s offline file tools, instead of the paper’s internal search.
GEPA: the paper does not publish its GEPA settings;
gepa/run.yamluses GEPA’s default reflection minibatch of 3, Pareto selection over the validation questions and 4 epochs.Codex harness: a Linux container per session with only the workspace and the corpus, and only the model provider reachable; Codex’s shell snapshot off; reasoning effort medium (SkillOpt’s Codex default leaves Codex’s own); 8 rollouts at a time (SkillOpt: 4).
Data: questions and answers from the latest dataset revision.
data.jsonlists the items whose sources changed since the paper’s revision: 6 test and 2 training questions for the results above. The results hold without them, except that in the Codex harness GEPA − no skill becomes +5.8 [+0.6, +11.2].ReHarness: OfficeQA is not a benchmark of the ReHarness paper (CL-Bench, MemoryBench, MemoryArena; Claude Opus 4.8, GPT-5.5 and Gemini 3.7 Flash), so its rows here are this repository’s settings, not a reproduction of a published number: GLM 5.3 Flash as every agent, SkillOpt’s initial skill as the text above the reflections, the venue’s acting pipeline as the decision agent, cosine ranking. The shared stream withholds the judge’s verdict (
feedback: false) that the paper’s environment outcome provides; the split run selects nothing, as the method has no validation step. The method page lists where the integration departs from the paper.
Hosted models and the dataset can change independently of the code, so the same commit need not give the same numbers.
Preserve the protocol#
Preserve slow_update_gate, trajectory_view, prompt_env, and final-selection settings (promote_final) when
comparing protocols; see SkillOpt for those controls and
Evaluation for reporting. Record the framework revision, the SkillOpt and GEPA commits the
dependency groups pin, the dataset revision and data.json’s drift list, the split, the seed skill, the model identifiers and
reasoning effort, the complete config, the harness and Codex version, and the concurrency. Evaluate no skill, the
seed skill and the trained skill in the same harness, and keep validation and test separate: do not select a
checkpoint on the test split. A smoke run is not an experiment. Reproduce a result
has a checklist.
For your own task, implement the agent and judge described in Task integration. For how the components exchange data, continue to Using and training adapters.