Self-Evolving Defense: Continual Security Policy Learning for LLM Agents

Le, Minh Nhat; Gondi, Nisarga; Peng, Yibo; Ni, Ronghao; Jia, Limin; Chen, Beidi; Zheng, Haizhong

Introduction

LLM agents read private data, call tools, and edit software, exposing them to direct jailbreaks, indirect prompt injections, and insecure code-generation attacks. Existing defenses typically cover one interface and remain fixed after deployment.

Self-Evolving Defense (SED) instead treats each judged failure as training data for an external policy memory. It converts harmful trajectories into reusable security policies and retrieves the relevant policies on later tasks, while keeping the agent's model weights frozen.

The SED loop: policies are retrieved into the agent's prompt, the agent acts, a judge scores the trajectory, and harmful episodes are synthesized into new policies for later episodes. — **Figure 1:** The SED loop. Memory is read during an episode and written only after an external judge scores the completed trajectory, so a successful attack can influence later episodes but never the one it appears in.

Why Static Defenses Don't Hold

Adaptive attackers revise their prompts after each failure, while static defenses retain nothing from prior attempts. The result is a defense that looks strong on a single shot and collapses once the attacker can iterate.

97.9%/93.4% AutoDAN-Turbo · Llama Guard 3 & SafeHarbor

39.9%/35.2% X-Teaming · Llama Guard 3 & SafeHarbor

This gap motivates a defense that updates from observed failures without retraining the agent.

Bar chart of attack success rate for AutoDAN-Turbo and X-Teaming against no defense, Llama Guard, and SafeHarbor. — **Figure 2:** Attack success rate of adaptive jailbreaks on HarmBench against an undefended agent and two static jailbreak defenses.

How SED Works

SED wraps a frozen agent with a policy memory that is read during an episode and updated only after the episode has been scored.

Retrieve select relevant policies

Rank stored policies for the task, select up to three across policy families, and include their covering parents.

Act run the frozen agent

Run the frozen agent with the retrieved policies appended to its system prompt.

Judge score the full trajectory

Score the full role-labeled trajectory; retain only harmful episodes and their supporting evidence.

Learn store and consolidate failures

Distill the failure into enforceable policies, then merge or organize them in the policy tree.

Auto-advances · hover to pause · click a step to jump

Experimental Evaluation

We evaluate three open-source models on eight chat, tool-use, and coding benchmarks. Each benchmark starts from an empty memory, evolves over its own stream, and uses its official grader.

Main results · DeepSeek-V4-Flash · 8 benchmarks, 5 defenses

SED leads six of eight security benchmarks, and the gains are not bought with refusals.

Compared against the undefended agent and the strongest static baseline on each benchmark. Attack success rate (ASR) is lower-is-better; utility is higher-is-better.

Security benchmarks led 6/ 8 lowest ASR of any defense on WJB, HB_dir, HB_adv, RedCode, AgentDojo, AgentDyn

RedCode risky execution 95.6%→11.3% specialized defenses leave it at 49% or above

AgentDojo utility under attack 44.4%→87.9% the agent still finishes the task; clean utility unchanged at 89.7%

SWE-bench resolve rate 62.6%→68.8% highest of all evaluated defenses; next best DRIFT at 67.6%

Attack success rate lower is better

WildJailbreak

31.4 Undefended

17.6 SafeHarbor

2.1 SED

RedCode

95.6 Undefended

49.3 SafeHarbor

11.3 SED

AgentDojo

49.0 Undefended

3.7 DRIFT

0.42 SED

AgentDyn

33.0 Undefended

2.1 DRIFT

0.54 SED

Benign utility higher is better

AgentDojo under attack

44.4 Undefended

78.2 SafeHarbor

87.9 SED

SWE-bench resolve rate

62.6 Undefended

67.6 DRIFT

68.8 SED

AgentDyn under attack

49.1 Undefended

66.4 SafeHarbor

69.3 SED

Undefended Strongest baseline on that benchmark SED

**Table 1: Attack success rate (%), lower is better.** DeepSeek-V4-Flash across eight benchmarks. WJB = WildJailbreak, HB = HarmBench (direct and adversarial), RC = RedCode, AgH = AgentHarm, ADojo = AgentDojo, ADyn = AgentDyn.
Defense	WJB	HB_dir	HB_adv	RC	AgH	FCV	ADojo	ADyn
Undefended	31.40	25.94	25.55	95.60	42.60	19.17	49.00	33.04
Llama Guard 3	29.31	2.81	8.16	89.12	3.00	20.43	49.32	29.11
SafeHarbor	17.55	15.00	9.64	49.28	1.59	5.71	11.90	10.18
DRIFT	23.90	24.06	16.64	91.12	31.06	1.48	3.69	2.14
GuardAgent	35.65	23.44	18.16	82.40	18.34	6.19	12.60	9.82
SED	2.10	0.94	3.07	11.30	6.20	2.91	0.42	0.54

**Table 1b: Benign utility (%), higher is better.** DeepSeek-V4-Flash. ADojo = AgentDojo, ADyn = AgentDyn; u = benign utility, ua = utility under attack, res = SWE-bench resolve rate.
Defense	AgH	ADojo_u	ADojo_ua	ADyn_u	ADyn_ua	FCV_res
Undefended	67.30	89.69	44.36	73.33	49.11	62.60
Llama Guard 3	57.50	92.78	42.36	65.00	45.36	55.80
SafeHarbor	41.83	89.69	78.19	71.67	66.43	56.00
DRIFT	76.73	86.60	76.08	28.33	28.21	67.60
GuardAgent	75.85	73.20	58.90	53.33	43.04	38.80
SED	63.10	89.69	87.88	73.33	69.29	68.80

Prompt Injection on DTap

DTap evaluates indirect prompt injection against CRM, workflow, and coding agents on three models. Figure 3 places every defense on the security–capability plane; the card below gives the per-domain numbers.

Model Y-axis

Macro-average indirect attack success (%) across CRM, workflow, and code

Model outer ring

Defense corner chip; no-defense shows the model only

Ideal defenses sit in the lower-right: high benign utility, low attack success. Hover a badge to zoom and focus it; nearby points fade back. SED badges cluster toward the ideal corner across models.

Figure 3: Security versus capability on DTap. Macro-average attack success (toggle indirect/direct) against benign task success across CRM, workflow, and code. Colored ring = model; corner chip = defense (same badges as the paper figure). Overlapping badges are nudged apart; thin lines mark true data positions. On DeepSeek-V4-Flash, SED lowers indirect attack success from 79–90% to 19–26%, a 60- to 65-point drop in every domain, while benign task success moves by less than 1.2 points.

DTap · 3 models · CRM, workflow, and code agents

Indirect injection success drops by 20 to 65 points in every domain, on every model, while benign task success stays essentially unchanged.

Headline number is indirect-injection ASR, undefended → SED. Direct-injection ASR and benign task success are shown underneath each chip.

DeepSeek-V4-Flashmost exposed undefended

CRM −65 pts

86.7→21.5%

direct 96.7 → 22.5 · benign 87.1 → 86.6

Workflow −60 pts

78.9→19.2%

direct 83.1 → 31.2 · benign 91.3 → 90.2

Code −64 pts

89.7→26.1%

direct 95.0 → 24.0 · benign 99.7 → 99.1

GLM-5.2

CRM −33 pts

41.3→8.3%

direct 32.2 → 12.2 · benign 72.1 → 75.8

Workflow −23 pts

32.4→9.6%

direct 47.3 → 17.3 · benign 92.3 → 91.8

Code −46 pts

49.7→3.7%

direct 57.9 → 2.5 · benign 97.9 → 97.9

Kimi K3

CRM −38 pts

41.3→3.3%

direct 42.2 → 7.8 · benign 88.5 → 88.2

Workflow −20 pts

25.8→5.9%

direct 26.3 → 10.4 · benign 89.2 → 88.2

Code −34 pts

36.6→2.3%

direct 51.3 → 4.4 · benign 99.6 → 99.6

**Table 2: DTap prompt-injection results (%).** ind and dir are indirect and direct injection ASR (lower is better); ben is benign utility (higher is better).
Defense	CRM			Workflow			Code
Defense	ind ↓	dir ↓	ben ↑	ind ↓	dir ↓	ben ↑	ind ↓	dir ↓	ben ↑
DeepSeek-V4-Flash
Undefended	86.67	96.67	87.12	78.85	83.12	91.34	89.70	95.04	99.70
Llama Guard 3	66.67	67.42	82.42	54.81	45.45	92.12	43.64	41.67	98.48
SafeHarbor	58.67	80.00	39.39	62.50	67.53	88.06	83.64	68.60	91.21
DRIFT	33.33	83.33	86.06	57.69	72.73	92.81	20.86	90.91	99.09
GuardAgent	48.32	88.89	80.00	66.35	64.94	90.70	27.95	92.56	99.39
SED	21.48	22.47	86.59	19.23	31.17	90.15	26.06	23.97	99.09
GLM-5.2
Undefended	41.33	32.22	72.12	32.38	47.30	92.33	49.70	57.85	97.88
Llama Guard 3	18.67	17.78	75.15	8.74	32.00	93.94	47.56	51.67	98.78
SafeHarbor	12.00	13.33	58.18	26.00	28.00	92.60	19.39	28.10	98.78
DRIFT	10.67	20.00	75.76	21.20	37.30	93.20	14.55	43.80	98.78
GuardAgent	36.00	32.22	73.33	30.50	50.70	90.50	36.65	53.72	98.18
SED	8.26	12.20	75.78	9.62	17.33	91.78	3.66	2.50	97.87
Kimi K3
Undefended	41.33	42.22	88.48	25.81	26.32	89.22	36.59	51.28	99.62
Llama Guard 3	21.33	34.44	88.48	8.08	16.88	93.67	33.94	43.48	100.00
SafeHarbor	15.33	17.78	70.30	4.30	11.69	91.64	9.76	30.25	98.18
DRIFT	16.67	30.00	87.88	21.43	30.67	93.41	9.09	40.68	99.62
GuardAgent	25.33	41.11	81.82	19.35	28.57	87.58	18.18	46.90	100.00
SED	3.33	7.78	88.18	5.88	10.39	88.18	2.28	4.39	99.63

Holding Up Under Adaptive Attack

On DeepSeek-V4-Flash, SED lowers AutoDAN-Turbo and X-Teaming success from 98.0% and 50.0% to 17.8% and 7.8%. The same advantage remains on GLM-5.2 and Kimi K3.

**Table 3: Attack success rate (%) under adaptive attack, lower is better.** NoDef = undefended, LG = Llama Guard 3, SH = SafeHarbor, DR = DRIFT, GA = GuardAgent.
Attack	NoDef	LG	SH	DR	GA	SED
DeepSeek-V4-Flash
AutoDAN-Turbo	98.00	97.92	93.44	79.70	87.80	17.82
X-Teaming	50.00	39.90	35.21	50.73	47.40	7.81
GLM-5.2
AutoDAN-Turbo	30.31	29.25	23.12	27.81	38.44	7.10
X-Teaming	15.62	14.90	10.10	13.12	15.94	8.50
Kimi K3
AutoDAN-Turbo	19.38	16.56	12.19	19.06	19.18	10.43
X-Teaming	28.13	15.58	13.90	20.76	20.92	10.51

AutoDAN-Turbo jailbreaks 98 of 100 undefended behaviors at a median of 3 iterations. Against SED, it succeeds on 10 of 100 and requires a median of 11.5 iterations for those successes.

Heatmap of attack iterations to jailbreak for each defense, one column per behavior and one row per attack iteration. — **Figure 4:** Attack iterations to jailbreak. AutoDAN-Turbo against DeepSeek-V4-Flash, one panel per defense. Each column is one of 100 behaviors in attacker order; each row is an attack iteration. Color is the judge score, light green (safe) to dark red (jailbroken); white marks iterations after the attack has already succeeded and stopped.

Where Does the Robustness Come From?

We test whether the gains come from extra prompt text or from the judge alone.

**Table 4: Memory ablation on DeepSeek-V4-Flash.** Attack success rate (%), lower is better. Each row adds one component to the undefended agent.
Variant	AutoDAN-Turbo	X-Teaming	RedCode
No memory (undefended)	98.00	50.00	95.60
+ episodic recall	42.27	19.69	25.60
+ policy synthesis	28.12	14.00	13.70
+ consolidation (full SED)	17.82	7.81	11.30

Episodic recall already cuts attack success, policy synthesis lowers it further, and tree consolidation yields the full result. Because synthesis and consolidation compress the retrieved text, prompt length does not explain the ordering. GuardAgent also uses a model for safety decisions but remains at 87.8% and 47.4% under the two adaptive attacks, showing that the judge alone is insufficient.

Freezing the memory after 100 behaviors lowers cumulative ASR from 23.4% to 13.4%; continued evolution reaches 9.4%. Applying the frozen attack-derived library to disjoint benign data costs at most 2.2 utility points.

**Table 5: Benign utility under policy transfer (%), higher is better.** A frozen policy library learned on harmful episodes is applied read-only to a disjoint benign split.
Benign benchmark	No memory	Attack-derived policies
WildJailbreak benign	97.60	96.00
AgentHarm benign	66.90	64.68

Cumulative attack success over an X-Teaming stream, comparing the undefended agent, SED frozen after 100 behaviors, and SED evolving continuously. — **Figure 5:** Frozen versus evolving policy memory. Cumulative X-Teaming attack success against a GLM-5.2 agent on HarmBench.

BibTeX

@inproceedings{le2026sed,
  title     = {Self-Evolving Defense: Continual Security Policy Learning
               for LLM Agents},
  author    = {Le, Minh Nhat and Gondi, Nisarga and Peng, Yibo and Ni, Ronghao
               and Jia, Limin and Chen, Beidi and Zheng, Haizhong},
  year      = {2026},
  note      = {Preprint},
  url       = {https://infini-ai-lab.github.io/SED/},
}