💬 NLP

A Self-Evolving Multi-Agent Framework Defense against LLM Jailbreak Attacks

This paper proposes a self-evolving, parameter-free multi-agent defense framework that utilizes a persistent cross-interaction rule memory to abstract successful jailbreak attempts into generalizable method-level rules, enabling LLMs to dynamically adapt to evolving attack strategies without compromising benign utility or increasing over-refusal.

Original authors: Tongyan Hu, Bryan Hooi

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tongyan Hu, Bryan Hooi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Large language models are powerful tools that can write stories, solve problems, and answer questions, but they have a significant weakness: they can be tricked into ignoring their safety rules. Imagine a highly trained assistant who has been taught never to give dangerous instructions, like how to build a weapon or bypass a security system. A clever trickster can still get this assistant to comply by wrapping the dangerous request in a complex disguise, such as pretending to be a character in a story or hiding the request inside a computer code. These tricks, known as "jailbreaks," exploit the model's desire to follow instructions, allowing harmful ideas to slip past its defenses. For years, the standard way to protect these models has been to set up a fixed barrier, like a static list of forbidden words or a rigid set of instructions programmed into the system. However, just as a thief can eventually pick a lock they have studied, attackers keep inventing new disguises that these static barriers cannot recognize, leaving the models vulnerable to evolving threats.

Researchers at the National University of Singapore have proposed a different approach to this problem, one that allows the defense system to learn and grow stronger over time. Instead of relying on a fixed set of rules that never changes, they built a system that treats every failed attempt to break the safety rules as a lesson. When the system encounters a new type of trick that successfully bypasses its defenses, it does not just block that single request and move on. Instead, it analyzes the structure of the trick, strips away the specific harmful topic, and creates a new, reusable rule that targets the method used to disguise the request. This new rule is stored in a persistent memory, much like a security guard writing down the specific uniform a thief wore, so that the guard can spot anyone wearing that same uniform in the future, regardless of what they are trying to steal.

The system operates through a team of four specialized components that work together during every interaction. First, a classifier looks at an incoming request to see if it is using a known trick. If the request is harmless, it is answered normally. If the request looks like a trick, the system checks its memory for a rule that matches the specific style of the disguise. If a matching rule exists, the system immediately applies a strict refusal or a safe alternative response. If the request is a brand-new type of trick that the system has never seen before, it is allowed to pass through initially. If the model then accidentally produces a harmful answer, a reflection agent steps in. This agent analyzes the failure, identifies the structural pattern of the attack, and writes a new rule into the memory bank. This new rule is then available to protect against all future requests that use the same structural pattern, effectively turning a single failure into a permanent shield against an entire family of similar attacks.

The researchers tested this self-evolving defense on several different models, including both open-source programs and powerful commercial systems that cannot be modified directly. They subjected these models to four distinct families of jailbreak attacks, which included methods like role-playing, code obfuscation, and multi-step manipulation. The results showed that as the system encountered more attacks and added more rules to its memory, its ability to stop them improved dramatically. In the early stages, when the memory was empty, the system was vulnerable, but after learning from just a few examples, it reduced the success rate of these attacks to near zero. Crucially, this learning happened without changing the underlying model's brain; the system simply became smarter about how it processed inputs by adding external notes to its memory.

A key finding of the study is that this method does not make the system overly cautious or rude to normal users. The researchers checked to see if the growing list of rules would cause the model to refuse harmless requests, a problem known as over-refusal. They found that even as the memory filled with rules from different types of attacks, the system remained accurate in distinguishing between dangerous tricks and safe questions. The system only refused requests that matched the specific structural patterns of the attacks it had learned to recognize. This suggests that the defense is precise, targeting the method of the attack rather than the topic of the conversation. The study also showed that the system remained robust even when attackers tried to combine different tricks to evade detection, indicating that the memory-based approach can adapt to complex, evolving threats.

This work represents a shift from static protection to dynamic adaptation. While previous defenses were like a locked door that could be picked if the right tool was found, this new framework acts more like a security team that updates its watch list after every breach. By converting failures into reusable knowledge, the system builds a defense that grows stronger the more it is tested. The researchers demonstrated that this approach works across different types of models and attack styles, offering a promising path toward making artificial intelligence safer in a world where attackers are constantly inventing new ways to bypass the rules. The study concludes that while no system is perfect, giving a model the ability to learn from its own mistakes in real-time provides a level of resilience that static defenses simply cannot achieve.

Technical Summary: A Self-Evolving Multi-Agent Framework Defense against LLM Jailbreak Attacks

1. Problem Definition

Large Language Models (LLMs) remain vulnerable to jailbreak attacks that exploit techniques such as role-playing, obfuscation, code transformation, and multi-step indirection to elicit harmful outputs. While existing defenses (e.g., static system prompts, supervised safety fine-tuning, RLHF, and external classifiers) are effective against known patterns, they suffer from a fundamental limitation: their safety behaviors are fixed at deployment. Consequently, they cannot accumulate defensive experience or adapt to unseen, compositional, or obfuscated strategies. This static nature creates a mismatch where adversaries can exploit novel patterns to bypass defenses that do not learn from past failures.

2. Methodology

The authors propose a self-evolving test-time defense centered on a persistent, cross-interaction rule memory. This framework operates without modifying the base model's parameters ( $\theta$ ), making it applicable to both open-weight and black-box API models.

Core Mechanism: Persistent Rule Memory

The system maintains an external memory $R_t$ that accumulates structured, method-level defense rules. Unlike previous self-reflection approaches that treat interactions independently or use transient context, this framework converts observed failures into reusable knowledge.

Rule Structure: Each rule $r_i$ $r_{i}$ takes the form: [label=L_i] If the request uses P_i, then D_i.
- $L_i$ : A method-level attack label (e.g., roleplay-nested-persona).
- $P_i$ : A description of the structural attack wrapper.
- $D_i$ : The refusal constraint.
Generalization: Rules abstract the structural wrapper rather than the specific harmful topic. Thus, a single rule induced from one instance generalizes across an entire attack family, and the label space expands automatically as structurally novel wrappers are encountered.

Framework Components

The framework is realized through four cooperating agents (prompted modules) that implement five operators:

Attack-Pattern Classifier ( $\phi$ ): Maps input $x^{(t)}$ to a method-level label. It returns benign, an existing label from memory, or other if the wrapper is novel.
Rule Triggering Agent (A1): Retrieves relevant rules from memory based on the classified label. It uses a three-stage matching process: exact label match, LLM-based relevance scoring, and lexical keyword fallback. It retrieves at most $K=2$ rules.
Policy Decision Agent (A2): Maps triggered rules to a response policy: hard-refuse (for high-risk inputs), soft-refuse, or allow.
Response Generation Agent (A3): Produces the final output $y^{(t)}$ by injecting the selected policy as a dynamic system instruction. In hard-refusal mode, it enforces strict constraints against operational details or code.
Self-Reflection Agent (A4): Analyzes the interaction post-generation. If a violation is detected (via an LLM-as-judge scoring the response on a 1–10 harm scale), it induces a new rule $r_{new}$ summarizing the attack wrapper and updates the memory $R_{t+1}$ via semantic deduplication and capacity bounds.

Inference Procedure

The process follows a sequential loop:

Classify input ( $\phi$ ).
Retrieve relevant rules ($match$).
Decide policy ( $g$ ).
Generate response ( $f_\theta$ ).
Detect violation ( $h$ ).
If a violation occurs on a non-benign input, induce a new rule ( $F$ ) and update memory ( $U$ ).

3. Key Contributions

The paper emphasizes that the primary contribution is the mechanism of memory-based adaptation, not the specific decomposition into agents.

Self-Evolving Defense: A persistent, cross-interaction rule memory that converts observed jailbreak failures into reusable, method-level defense rules, enabling continual adaptation without parameter updates.
Wrapper-Level Abstraction: A dynamic rule-triggering mechanism that abstracts attacks at the wrapper level. This allows a single induced rule to generalize across an attack family, with an automatically expanding label space for structurally novel attacks.
Black-Box Applicability: The method operates entirely through external memory and prompting, requiring no access to model weights or gradients, thus functioning on both open-source and proprietary API models.

4. Experimental Results

The framework was evaluated across four black-box jailbreak families (DeepInception, CodeChameleon, ReNeLLM, FlipAttack) and multiple models (Qwen2.5-7B, Llama3.1-8B, Gemini-3-Flash-Preview).

Attack Success Rate (ASR): The method substantially reduces ASR compared to baselines (No Defense, Defense Prompt, Self-Reminder, AutoDefense). For instance, on CodeChameleon attacks against Llama3.1-8B, the ASR-gpt dropped from 58.0% (No Defense) to 1.9% (Ours).
Benign Utility: The defense preserves utility on benign tasks (MMLU, GSM8K), with performance remaining within approximately two points of the undefended model.
Self-Evolving Analysis: In a cold-start setting (empty memory), the system learns defense rules during the first batch of attacks. Subsequent rounds show ASR dropping to near zero, demonstrating that the system generalizes from past failures to structurally similar future attacks.
Robustness: The framework remains robust under adaptive composite-wrapper attacks designed to evade rule triggering.
Over-Refusal: As the memory grows, over-refusal on benign prompts (measured via XSTest) remains low (rising only to 9.6% after accumulating rules), indicating that the system does not become overly restrictive.
Cost: The inference overhead is approximately 3.07× LLM calls and 1.43× latency compared to a single-shot model, comparable to multi-agent baselines like AutoDefense.

5. Significance and Claims

The paper claims that current safety mechanisms fail because they do not accumulate defensive experience over time. By shifting from static safeguards to a self-evolving defense system, the authors demonstrate that persistent test-time adaptation provides robustness gains unattainable by static defenses.

The significance lies in the ability to:

Generalize across families: A single rule covers an entire structural family of attacks, rather than requiring instance-specific corrections.
Adapt without retraining: The system updates its behavior based on past failures without requiring fine-tuning or white-box access.
Maintain utility: It achieves high robustness without significantly degrading performance on benign tasks or inducing excessive over-refusal.

The authors acknowledge limitations, including the current focus on single-turn, prompt-level attacks and the lack of explicit rule-pruning mechanisms for very long interaction streams. However, they posit that moving toward self-evolving systems is a promising direction for reliable LLM deployment.