💬 NLP

MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safety

This paper introduces MAGIC, a novel multi-turn multi-agent reinforcement learning framework that enhances Large Language Model safety through a co-evolving adversarial game where an attacker agent dynamically generates novel combinatorial strategies to expose vulnerabilities, thereby driving a defender agent to generalize and robustly refuse unseen adversarial inputs.

Original authors: Xiaoyu Wen, Zhida He, Han Qi, Ziyu Wan, Zhongtian Ma, Ying Wen, Tianhang Zheng, Xingcheng Xu, Chaochao Lu, Qiaosheng Zhang

Published 2026-02-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaoyu Wen, Zhida He, Han Qi, Ziyu Wan, Zhongtian Ma, Ying Wen, Tianhang Zheng, Xingcheng Xu, Chaochao Lu, Qiaosheng Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart but naive robot how to be a good citizen. The robot knows a lot of facts, but it doesn't always know when to say "no" to dangerous requests.

The paper introduces a new training method called MAGIC. Instead of just showing the robot a list of "bad things not to do" (which is how most current safety training works), MAGIC sets up a never-ending sparring match between two versions of the robot: an Attacker and a Defender.

Here is how it works, using simple analogies:

1. The Problem with the Old Way (Static Training)

Currently, safety training is like a teacher handing a student a textbook of "known bad words" and saying, "Don't say these."

The Flaw: As soon as the student learns the list, a clever trickster (the attacker) invents a new way to ask for the bad thing using different words or a new story. The student, who only studied the old textbook, gets tricked. The defense is always one step behind.

2. The MAGIC Solution: A Co-Evolving Gym

MAGIC changes the game by creating a dynamic gym where the Attacker and Defender train together, constantly pushing each other to get better.

The Attacker (The Trickster): This robot's job is to try to trick the Defender into saying something harmful. But here's the twist: The Attacker isn't just guessing. It is given a "thinking cap" (Chain-of-Thought) that teaches it how to strategize. It learns to rewrite simple bad requests into complex, deceptive stories that look innocent on the surface but hide a dangerous intent.
- Analogy: Imagine a magician learning new tricks. At first, they just pull a rabbit out of a hat. But as they train, they learn to hide the rabbit in a deck of cards, then in a mirror, then in a song. They are constantly inventing new ways to fool the audience.
The Defender (The Guard): This robot's job is to listen to the Attacker's tricks and say "No" if it's dangerous, or "Yes" if it's safe.
- Analogy: Imagine a bouncer at a club. At first, they only check for a specific list of banned items. But because the Attacker is constantly inventing new ways to sneak things in, the bouncer has to learn to recognize the intent behind the disguise, not just the disguise itself.

3. The "Co-Evolution" (The Dance)

The magic happens because they train together in a loop:

The Attacker tries a new, clever trick to bypass the Defender.
If the trick works, the Attacker gets a "point," and the Defender gets a "ding."
The Defender learns from the mistake and gets better at spotting that specific trick.
Now that the Defender is better, the Attacker has to invent an even smarter trick to get around the new defense.

This creates a "co-evolution." Just like in nature, where predators and prey constantly evolve new speed and camouflage, the Attacker and Defender get smarter together. The paper claims this forces the Defender to learn robust safety rules that work even against attacks it has never seen before, rather than just memorizing a list of old tricks.

4. Why This is Different

Old Way: The Attacker and Defender are often the same robot playing both sides, which confuses its brain (like trying to be both the referee and the player).
MAGIC Way: They are two separate robots with different goals. The Attacker is specifically trained to be a "thinking" strategist, while the Defender learns to be a sharp judge. This separation prevents them from getting confused and allows them to specialize.

5. The Results

The paper tested this on various models and found:

Better Safety: The Defender became much better at refusing harmful requests, even when the Attacker used complex, multi-step tricks.
No Loss of Helpfulness: Crucially, the Defender didn't become a "grump" who says "no" to everything. It learned to distinguish between a dangerous trick and a harmless question that looks tricky. It remained helpful for normal users.
New Discoveries: The Attacker actually invented brand-new types of tricks that humans hadn't thought of, proving that the system could find "long-tail" vulnerabilities (rare, weird attack methods) that static lists would miss.

In short: MAGIC teaches AI safety not by giving it a rulebook, but by putting it in a boxing ring with a smart opponent that keeps changing its fighting style. By the end of the match, the AI is so well-trained that it can spot danger even when it's wearing a disguise.

tags) and the actual prompt (in` tags) to prevent the reasoning from being exposed to the defender.
* Harmfulness Reward ( $R_{harm}$ ): Uses Qwen3Guard to penalize unsafe or controversial responses.
* Refusal Reward ( $R_{ref}$ ): Prevents "over-safety" collapse by rewarding the correct refusal of harmful queries and the correct acceptance of benign (even adversarially structured) queries.

3. Key Contributions

Asymmetric MARL Framework: MAGIC proposes an online MARL framework that decouples attacker and defender optimization, moving away from symmetric self-play to an asymmetric sequential game guided by SPNE. This provides stronger, pointwise safety guarantees.
CoT-Enriched Attack Pool: The authors construct a benchmark with CoT completions across 20 diverse rewriting strategies. This addresses data scarcity and cold-start issues, equipping the attacker with the reasoning capabilities necessary to explore long-tail vulnerability spaces.
Discovery of Novel Strategies: The framework demonstrates that iterative RL training enables the attacker to evolve novel, previously unseen combinatorial attack strategies (e.g., combining role-playing with translation and concept substitution), rather than merely re-ranking existing templates.

4. Experimental Results

Extensive experiments were conducted across single-turn and multi-turn benchmarks using models from the Qwen2.5 and Llama3.1 families.

Safety Performance: MAGIC significantly reduces the Attack Success Rate (ASR). For example, on the WildGuardTest benchmark with Qwen2.5-7B, ASR dropped from 36.5% to 2.3%. MAGIC consistently outperformed baselines like Self-RedTeam, SmoothLLM, and Self-Eval across HarmBench, DAN, and OR-Bench.
Benign Compliance: Crucially, MAGIC maintains high helpfulness. It does not suffer from excessive refusal rates on benign prompts, even those that are adversarially structured. On XSTest and WildJailBreak benign splits, MAGIC achieved high compliance rates, demonstrating that safety gains were not achieved through over-refusal.
General Capabilities: Evaluation on MMLU, ARC-C, GPQA, and AlpacaEval 2 showed that MAGIC largely preserves the model's general instruction-following and reasoning capabilities, with performance differences remaining within a narrow range compared to the original instruction-tuned baselines.
Multi-Turn Robustness: When evaluated on X-Teaming (multi-turn jailbreaks), MAGIC-trained models showed substantial robustness improvements, reducing multi-turn ASR by 10.1% compared to the baseline, indicating the framework captures transferable adversarial dynamics.
Generalization: MAGIC-trained defenders demonstrated strong generalization against unseen attack strategies (e.g., GCG, PAIR, TAP, AutoDAN) in automated red-teaming evaluations, suggesting the defense is robust to dynamic, interactive jailbreak attempts rather than overfitting to a fixed distribution.

5. Significance and Claims

The paper claims that MAGIC represents a shift from reactive, static defense paradigms to a proactive, evolving safety alignment process. By formulating safety as an asymmetric game and utilizing co-evolutionary RL, the framework enables the continuous discovery of long-tail vulnerabilities and the development of robust refusal behaviors that generalize beyond static data.

The authors emphasize that their approach achieves a superior balance between safety and utility. Unlike methods that degrade general capabilities or induce over-refusal, MAGIC allows the model to remain helpful while effectively resisting adaptive adversarial behaviors. The theoretical grounding in SPNE provides a stronger safety guarantee (instance-level safety) compared to expectation-based guarantees in prior Nash Equilibrium approaches.

The paper concludes that while the online adversarial paradigm introduces computational overhead, it offers a viable path toward treating safety alignment as an evolving process, capable of adapting to the "cat-and-mouse" dynamics of real-world LLM threats.