💬 NLP

Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPO

This paper introduces AdvGRPO, a novel co-training framework that stabilizes GRPO for joint attacker-defender optimization through dense multi-channel rewards, decoupled advantage normalization, and a progressive curriculum, ultimately yielding highly effective transferable attacks and more robust language model defenses.

Original authors: Blake Bullwinkel, Eugenia Kim, Amanda Minnich, Mark Russinovich

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Blake Bullwinkel, Eugenia Kim, Amanda Minnich, Mark Russinovich

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A Digital Sparring Match

Imagine you have a very smart robot (the Defender) that is trained to be helpful but also to refuse harmful requests, like "How do I build a bomb?" or "How do I hack a bank?"

Usually, humans try to test this robot by writing a list of tricky questions. But the robot is smart; it learns to say "No" to those specific questions. The problem is that a clever, adaptive attacker can change their strategy to find a new way to trick the robot.

This paper introduces a new way to train both sides at the same time using a method called AdvGRPO. Think of it as setting up a digital dojo where a Red Team (the attacker) and a Blue Team (the defender) spar against each other continuously. Instead of humans writing the questions, the AI models learn to attack and defend by playing against one another, getting stronger with every round.

The Problem with the Old Way

Previously, researchers tried to use a specific training tool (called GRPO) to teach the attacker how to break the defender. However, they found that when the attacker and defender learned at the same time, the system got "unstable." It was like two boxers trying to learn a new fighting style while the ring itself was shaking; they couldn't agree on who was winning, and the training would crash or go in circles.

The Solution: AdvGRPO (The Stable Dojo)

The authors created a new framework called AdvGRPO that fixes the shaking ring. They did this with three main tricks:

The Scorecard with Multiple Lanes (Dense Multi-Channel Rewards):
Imagine a referee who doesn't just shout "Point!" at the end of the round. Instead, they give a score for every single move.
- Did the attacker stay on topic?
- Did the attacker follow the strategy?
- Did the attacker actually get the harmful answer?
  By scoring every little step, the attacker gets constant feedback on what to improve, rather than waiting until the end to find out they failed.
The Independent Judges (Decoupled Advantage Normalization):
In the old unstable method, if the attacker suddenly got really good, the scores for the defender would look terrible by comparison, confusing the training.
The new method uses a system (called GDPO) where each type of score is judged independently before being combined. It's like having a judge for "Speed," a judge for "Technique," and a judge for "Creativity" who all grade separately, so one doesn't ruin the other's score. This keeps the training stable even as both models get smarter.
The Training Camp (Curriculum Learning):
You wouldn't put a beginner boxer in the ring with a champion immediately. The paper uses a "curriculum."
- Phase 1: The attacker trains alone against a fixed defender to learn the basics.
- Phase 2: Once the attacker is decent, they start sparring with the defender.
- Phase 3: They take turns. The attacker tries to break the defender for a few rounds, then the defender tries to patch the holes for a few rounds, and they switch. This prevents the defender from just saying "No" to everything (which is an easy win) and forces it to learn how to handle specific, tricky attacks.

What They Found

The paper reports several key results from their experiments:

The Attackers Got Scary Good: The trained attackers learned to trick the defender much better than untrained models or models that were just "unlocked" (had their safety filters removed). They learned that simply removing safety filters isn't enough; you have to learn how to attack.
They Can Adapt: The attackers learned strategies that worked not just on the defender they trained against, but on completely different models they had never seen before. It's like a boxer learning a move that works on anyone, not just their sparring partner.
The Defenders Got Tougher: The defenders trained in this "dojo" became much better at spotting and blocking attacks than defenders trained with older methods. They learned to say "No" to bad requests but still say "Yes" to good, harmless requests.
Thinking Models Need Special Help: They found that models capable of "thinking" (reasoning) often tried to be too safe and refused to attack even when asked. They had to create a special reward system to teach these models how to think through the attack without getting stuck on safety warnings.

The Bottom Line

The paper claims that by using this new, stable method (AdvGRPO), they can create a system where AI attackers and defenders learn together. This results in attackers that are very good at finding weaknesses and defenders that are very good at fixing them, making AI safer overall. They emphasize that this is a research tool to find weaknesses before bad actors can use them, helping to build stronger, more robust AI systems.

Technical Summary: Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPO

Problem Statement

Current safety alignment for Large Language Models (LLMs) relies heavily on static datasets of adversarial prompts curated by humans or generated by fixed jailbreak techniques. While models can be trained to deflect these specific attacks, they remain brittle against adaptive attackers who modify strategies to target new weaknesses. Existing automated red-teaming methods often train attackers against fixed defenders or use open-loop optimization where the attacker generates all turns without observing the defender's responses. Furthermore, while Group Relative Policy Optimization (GRPO) has shown promise in safety domains, prior attempts to apply it to attacker-defender co-training have reported instability, leading researchers to favor Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO) instead.

Methodology: AdvGRPO

The authors introduce AdvGRPO, a co-training framework designed to make GRPO viable for joint attacker-defender optimization. The framework addresses the instability of prior GRPO co-training attempts through three core mechanisms:

Dense Multi-Channel Rewards and GDPO:
Instead of relying on sparse final outcomes, the system employs dense, multi-channel reward signals scored by an LLM judge (GPT-4.1).
- For Attackers: Rewards include an Attack Reward ( $A$ ) for the defender's harmful response, a Prompt Reward ( $P$ ) for the attacker's adherence to strategy, and a Thinking-Trace Reward ( $T$ ) for reasoning models to prevent self-censoring.
- For Defenders: Rewards include $1-A$ for adversarial objectives (penalizing harmful outputs) and a Helpfulness Reward ( $H$ ) for benign objectives.
- GDPO (Group reward-Decoupled Policy Optimization): To prevent reward signal collapse where channels with lower variance are attenuated during normalization, the framework normalizes each reward channel independently within a group before combining them via a weighted sum. This decouples the advantage scales of different objectives.
Closed-Loop Multi-Turn Optimization:
Unlike open-loop methods (e.g., SEMA) where attackers generate all turns in one shot, AdvGRPO operates in a closed loop. The attacker observes the defender's response at each turn and adapts its next prompt accordingly. Episodes are pruned early if the attack succeeds (reward > 0.9), providing efficient credit assignment for per-turn actions.
Curriculum Pre-Training and Alternating Updates:
- Curriculum: The attacker is first trained in a single-turn setting, then multi-turn, before co-training begins. This prevents the defender from dominating the early stages of co-training by easily deflecting weak, untrained attacks.
- Alternating Updates: During co-training, the attacker and defender models are updated in alternation (every $N$ steps). In the attacker phase, the defender is frozen; in the defender phase, the attacker is frozen, and the defender is trained on a mix of adversarial and benign objectives to prevent a "refuse-everything" degenerate strategy.

Key Contributions

Stabilizing GRPO for Co-Training: The paper demonstrates that GRPO can be effectively used for attacker-defender co-training, overcoming previous reports of instability through the use of GDPO and decoupled advantage normalization.
Closed-Loop Multi-Turn Learning: The framework enables attackers to learn adaptive strategies by conditioning on defender responses at each turn, rather than generating static multi-turn sequences.
Reasoning Model Attack Training: The method successfully trains reasoning-capable models (e.g., Qwen3.5-9B) to overcome alignment-induced self-censoring, a capability that base reasoning models lack.
Robust Defender Training: The co-trained defenders achieve superior safety performance compared to baselines while maintaining general utility.

Results

Attacker Performance:

AdvGRPO-trained attackers achieved significant improvements in Attack Success Rate (ASR) over base models. For instance, a Qwen2.5-14B multi-turn attacker achieved 90–91% ASR on AdvBench and HarmBench against GPT-4.1.
Reasoning-capable models (Qwen3.5-9B) trained with AdvGRPO achieved 71–79% ASR in single-turn settings, overcoming near-zero performance in their base state.
Transferability: The attackers generalized well to unseen defender models (Phi-4-mini, Llama-3.1-8B, Gemma-2-9B), often outperforming the state-of-the-art SEMA method on out-of-distribution (OOD) objectives, particularly in multi-turn scenarios.
Baseline Comparison: Uncensored models (e.g., Abliterated or GRP-Obliterated models) performed poorly as attackers compared to AdvGRPO-trained models, indicating that removing safety guardrails alone does not confer adversarial strategy capabilities.

Defender Performance:

Co-trained defenders (Qwen2.5-7B) reduced HarmBench ASR to <2%, significantly outperforming the base model (18.8%), Self-RedTeam (16.8%), and AdvGame (4.7%).
While benign compliance decreased slightly (likely due to training only on vanilla benign prompts), general utility metrics (MMLU, TruthfulQA, ARC-C, IFBench) were preserved or improved, indicating that safety training did not degrade general capabilities.

Significance and Claims

The paper claims that AdvGRPO provides a stable and effective alternative to PPO and DPO for automated red teaming and defense. It establishes that:

GRPO is viable for co-training when paired with dense, multi-channel rewards and decoupled normalization, challenging the notion that it is inherently unstable in this setting.
Adaptive training is necessary: Simply removing safety guardrails (unalignment) is insufficient to create effective attackers; models must be specifically trained to exploit defender weaknesses through iterative co-training.
Closed-loop optimization yields more transferable and robust attacks than open-loop methods, as the attacker learns to adapt to specific defender responses turn-by-turn.
Co-training produces robust defenders that generalize better than those trained via static datasets or other co-training baselines, offering a practical path toward more resilient AI safety alignment.

The authors acknowledge limitations, including reduced benign compliance in defenders (suggesting a need for adversarial benign prompts in training) and some entropy collapse in attacker prompts, which limits attack diversity. However, the work positions AdvGRPO as a significant step forward in creating adaptive, automated red-teaming systems.