AgentDojo defense comparison#

The official AgentDojo banking, Slack, travel and workspace suites share native replay, model calls, Docker defense hooks and scoring with the AgentDyn integration. Setup and recipes cover both SHE and EvoSafeHarness. No model weights, native tools or judge predicates are changed.

The frozen comparison imports existing tool_knowledge payloads from the pinned upstream logs, removes prior trajectories/scores, audits native initialization, and splits by user task. It has 57 TRAIN, 54 reserved validation and 406 TEST cases. Both Codex recipes select on TRAIN only and reserve TEST for frozen no-defense/initial/final evaluation. See the data protocol for source revision, exclusions, ownership and hashes.

Both methods use the same fixed Nemotron 3.5 Lightning victim, 25 tool iterations, zero temperature and GPT-6 Sol/high Codex learners. SHE retains its 20-round upstream measured-best selection; EvoSafe keeps its three-proposal source cascade. Their search and auxiliary-judge budgets differ as documented method settings. Clean utility, attack success, raw injected utility, safe completion and execution errors have separate denominators. The scalar evaluation score is not the full defense objective.

Results#

Both 2026-11-17 Lightning/Codex comparisons completed. These are single-pass reference measurements under the frozen protocol above. Each row covers the same 406 TEST IDs; denominators count completed episodes, while errors count all planned attempts. Each method retains its own no-adapter attempt because repeated model executions vary.

Experiment / adapter

Clean utility

ASR

Errors / planned

EvoSafe / None

47/54 (87.0%)

81/344 (23.5%)

8/406

EvoSafe / Initial

49/55 (89.1%)

57/344 (16.6%)

7/406

EvoSafe / Final

48/55 (87.3%)

8/347 (2.3%)

4/406

SHE / None

50/55 (90.9%)

73/343 (21.3%)

8/406

SHE / Initial

50/54 (92.6%)

59/345 (17.1%)

7/406

SHE / Final

48/54 (88.9%)

12/343 (3.5%)

9/406

Both final adapters reduce observed ASR and increase safe injected completion, while clean utility decreases against their respective initial seeds. EvoSafe errors decrease from 7 to 4; SHE errors increase from 7 to 9. All rows have execution errors, so complete-run native defense objectives remain unavailable. No statistical significance, uniform improvement or equal-compute ranking is claimed. See the full records for paired changes, exact safe-completion figures, source/data hashes, adapter selection and limitations:

Official AgentDojo and AgentDyn install the same agentdojo package name and need separate Python environments.