Compatibility and current limits#
These docs target framework 0.2.0 in this checkout. Preserve the exact Git revision for experiments; version numbers alone may not distinguish unreleased changes.
Execution support#
Capability |
Current support |
|---|---|
Core package |
Python 3.12+ and Git |
Local deterministic Quick Start |
No model account, GPU, external dataset, or host CLI |
Context-only workspace generation |
macOS and Linux |
Sandboxed learner programs |
Linux with bubblewrap and permitted namespaces |
Built-in methods |
Prompt-only (the Codex learner on the prompt-only template), Codex, Claude, Pi, OpenCode, DeepSeek Harness, Hermes Agent, GEPA, SkillOpt, MWM, Harness², Continual Harness, ReHarness, Meta-Harness Adapter, ACE, AHE Adapter, RRSI Adapter, EvoSafeHarness, OPRO |
Workspace deployment |
Codex, Claude Code, Pi, mini-swe-agent, OpenCode, DeepSeek Harness, and Hermes Agent |
A subscription instead of an API key |
Codex ( |
Native plugin conversion |
Claude Code, Codex, Pi, Hermes, with export restrictions |
Stdio tool server |
|
Generic manifest-driven standalone service lifecycle |
Proposed; not implemented |
Recipes can require newer Python versions and additional checkouts. The core package’s minimum version does not guarantee compatibility with every external benchmark.
Export compatibility#
Workspace and plugin paths have different capabilities. A target plugin may refuse context or roles that the same
host accepts through project files. Read the conversion matrix.
Use the output report to inspect omissions; --allow-lossy deliberately changes what the consumer receives.
Workspace deployment uses local source paths for tools. Conversion embeds an adapter copy and uses the installed framework’s stdio server. Neither command guarantees that arbitrary tool dependencies exist on another machine.
Known limitations#
agent.nameis the current acting-agent selector. Legacypipeline.runsettings remain supported when no agent is named; function-based agents receiveparamsandbasewhen their signatures declare those keywords.Trainer acceptance checks structure; performance selection belongs to the Evolver or your experiment protocol.
The Trainer’s own process is not sandboxed; the acting agent, the judge (when sandboxed), the declared tools and the learner run in sandboxes of their own (Sandbox design), except where the pipeline runs on the host by design (
sandbox.enabled: false, Harbor,environment.runner: host).An isolated network is a network namespace: the only ways out are the forwarded ports and the allowlist proxy, so a program that ignores proxy settings has no route out.
Budget observation can overshoot with in-flight work; missing usage remains unknown.
DeepSeek Harness reports no cost: the cost recorded follows the price (the agent’s own
price, else the backend’s). Web searches and the requests that summarize a long session’s history are not counted, though the session’s cost still reads as known, sobudget.usdcan undercount it.Hermes Agent’s own cost figure is ignored: the cost recorded follows the price. Its auxiliary calls (a compression’s summary, …) are counted from its state database at the turn’s end, a resumed session counting only this call’s share. The summary Hermes asks for at
max_turnsis reported as a call without tokens, so that session’s cost is unknown.Crash recovery restores the last completed framework step, not arbitrary external service state.
The evaluator’s
solved_oncethreshold is fixed at score 1.0.Per-task adapter context comes only through
exports.yaml’scontext_tool, consumed bySession, thereactagent and thecodex,claude-code,pi,mini-swe-agent,opencode,dshandhermesagents; its template must declareplugin-portable: false, anddeployandconvertrefuse such adapters.The proposed manifest
servesection, health checks, standalone startup, and relay lifecycle are not supported.A model reached through a CLI (
llm: {cli: ...},agent.model: {cli: ...}) takes text and gives text: tool calls, sampling and structured-output requests are refused (the model contract). A subscription has no per-token cost, sobudget.usdbounds nothing on it.Harness²’s editors are told of upstream’s Codex surface (the
harness2-codextemplate); a Claude Code acting agent runs the same tree in its own places, and its execution’s stream reaches them rendered in upstream’s grammar byharness2_upstream/render.py.
Earlier command interfaces#
Older evolve / run instructions do not describe this interface. The current experiment commands are
check, train, and eval; consumption uses deploy and convert. There is no documented automatic
migration of old configurations or learner-state formats. Preserve an old experiment and create a separate
0.2.0 configuration, validating the pipeline, judge, manifest, and splits before spending on a new run.