Continual Harness#

Continual Harness (Karten et al., 2026, code) keeps a harness H = (p, G, K, M), the base policy, sub-agents, skills and memory, and refines it while the agent acts: a Refiner reads a window of the agent’s own trajectory, identifies failure signatures in it and emits per-component edits that are applied in place, H <- H + delta, with no accept/reject gate and no score. Four passes per generation, one model call each: the policy is rewritten whole; sub-agents, skills and memory are edited as create / update / retire deltas on named entries; the memory pass is conservative, because the agent writes memory itself while acting.

Setup#

template: continual-harness@1.0
data: {train: train.txt, step_size: 3}                              # the Refiner's window: a step's rollouts
agent: {name: ../utils/agent/codex_agent.py:OfficeQACodexAgent}      # an agent that runs `codex exec` with the tree deployed
evolver:
  name: continual_harness
  params: {feedback: true}
llm: {model: <backend>/<model>}                                     # the Refiner's model

The template is the harness state in the layout Codex reads from its workspace root: AGENTS.md (p; ADAPTER.md is its copy), sub_agents/<name>/AGENT.md (G, installed as Codex agent roles), .agents/skills/<name>/SKILL.md with executable helpers under scripts/ (K) and memory/NOTES.md (M, notes under ## <topic> headings that the acting agent reads and writes during a task). The acting agent deploys the tree into its workspace and hands the memory it wrote back with its trajectory (codex_agent.py).

The step#

  1. The Trainer rolls the current harness out on the step’s tasks.

  2. The Refiner reads each trajectory (a Codex event stream, or a message list), its final output and status, and extracts failure signatures: a session that erred or timed out, a command that exited non-zero, an answer that never came; with feedback: true (the default) the judge’s verdict on each task scored below 1 is shown too, as a signal to act on, not a score to optimise. With feedback: false the Refiner sees the trajectories alone, the paper’s own setting, where no judge exists.

  3. Notes a session left in memory/NOTES.md are folded into the harness’s memory (a known topic is replaced, a new one appended).

  4. Four passes, in order, each one call of the run’s llm tool: the policy (AGENTS.md, rewritten whole, at most 8,000 characters), the sub-agents (create / update / retire), the skills (add / update / retire, with helpers) and the memory (at most memory_max_changes notes). Each pass is applied to the adapter and checked with check_spec; a pass whose result fails the check is dropped, the others stand.

Nothing is scored and nothing is selected: the harness after the last step is final. state/continual_harness.json logs every step; each step’s output/report.md says what each pass changed.

Parameters#

description: Continual Harness's Refiner (Karten et al. 2026, arXiv:2605.09998; sethkarten/continual-harness) as an
  Evolver on the `continual-harness` template. Each step it reads the step's trajectories (Codex event streams or
  message lists), extracts failure signatures (sessions that erred or timed out, commands that failed, answers that
  were missing, and, when `feedback` is on, the judge's verdicts), folds the notes the acting agent wrote into the
  memory, and runs four refinement passes with the run's `llm` tool, one per component - the base policy
  (AGENTS.md, rewritten whole), the sub-agents (sub_agents/, create / update / retire), the skills (.agents/skills/,
  add / update / retire, with executable helpers) and the memory (memory/NOTES.md, a few conservative changes). No
  score is optimised and no validation gate is applied - every pass is applied in place, as the paper's reset-free
  loop does; a pass whose result fails the template check is dropped, the others stand.
class: evolver.py:ContinualHarnessEvolver
needs: []
params:
  components: [policy, subagents, skills, memory]   # the passes to run, in this order
  feedback: true                 # show the judge's feedback to the Refiner (off: trajectory-only, as the paper's setting)
  trace_chars: 12000             # characters of each trajectory shown to the Refiner (the middle is cut)
  task_info_chars: 3000          # characters of the task description shown to the Refiner
  memory_max_changes: 5          # the memory pass changes at most this many notes
  min_signals: 0                 # skip the step when fewer failure signatures than this and no note arrived (0: never skip)
  max_tokens: 32000
  temperature: null              # null: the endpoint's default

What it is not#

The paper’s second loop, co-learning, updates the model’s weights from a teacher’s process rewards; it is not part of this Evolver. The paper’s Refiner sees no external feedback; feedback: true is a decision of this port that puts Continual Harness on the same label footing as GEPA and SkillOpt (the training questions’ verdicts), the gate being what still sets it apart. Harness² (harness2.md) uses no labels at all.

Example#

examples/officeqa/continual_harness/: OfficeQA in the SkillOpt paper’s setting, Codex as the acting agent, GLM 5.3 Flash acting and refining. Its stream_codex.yaml is the paper’s own protocol (one stream over the test questions, feedback: false): on OfficeQA’s 172 test questions in the same order as the Harness² stream it scores 0.634, against 0.576 for the initial skill it starts from (+5.8 [−1.2, +12.8] paired) and 0.651 for Harness² (−1.7 [−7.6, +4.1]), at about a quarter of Harness²’s cost (the example’s result, the recipe’s Flash row).