🤖 AI

Safety Alignment of LMs via Non-cooperative Games

This paper introduces AdvGame, a novel safety alignment paradigm that frames the interaction between an Attacker and a Defender language model as a non-zero-sum game trained via online reinforcement learning with preference-based rewards, resulting in a more robust and helpful Defender alongside a powerful general-purpose red-teaming agent.

Original authors: Anselm Paulus, Ilia Kulikov, Brandon Amos, Rémi Munos, Ivan Evtimov, Kamalika Chaudhuri, Arman Zharmagambetov

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anselm Paulus, Ilia Kulikov, Brandon Amos, Rémi Munos, Ivan Evtimov, Kamalika Chaudhuri, Arman Zharmagambetov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart but inexperienced robot (the Defender) how to handle a chaotic world full of tricky questions, some of which are dangerous.

Traditionally, developers tried to teach this robot by showing it a list of "bad questions" and saying, "Don't answer these." But the robot is smart; it eventually learns to memorize the list but fails when someone asks a new kind of bad question it hasn't seen before. It's like teaching a security guard to recognize only the specific faces on a "wanted" poster, rather than teaching them to spot suspicious behavior in general.

This paper introduces a new way to train the robot called AdvGame. Instead of a teacher and a student, they set up a video game with two players:

The Attacker (The "Trickster"): A robot whose only job is to invent new, clever ways to trick the Defender into saying something harmful.
The Defender (The "Guardian"): A robot whose job is to answer helpful questions while safely dodging the Trickster's traps.

The Core Idea: A Never-Ending Dance

In the old method, the Trickster would try to break the Guardian, then the Guardian would get patched up, then the Trickster would try again. It was a slow, back-and-forth "cat-and-mouse" game.

In AdvGame, they play simultaneously.

The Trickster tries to find a new hole in the Guardian's armor.
The Guardian instantly learns to patch that hole.
Because they are playing at the same time, the Guardian learns to handle any new trick the Trickster comes up with, not just the ones it saw yesterday.

The paper argues that this isn't a "zero-sum" game (where one wins and the other loses). Instead, it's a non-zero-sum game. The Trickster isn't trying to destroy the Guardian; it's trying to make the Guardian better by finding its weaknesses. The Guardian isn't trying to silence the Trickster; it's trying to stay helpful while refusing to be tricked.

The Scoreboard: "Better Than" vs. "How Many Points"

A major innovation in this paper is how they judge the players.

Old Way (Point-wise): A judge gives a score like "7 out of 10" to a single answer. This is tricky because "7" is subjective. The robot might learn to game the system by giving answers that look good to the judge but aren't actually safe (like a student memorizing the answer key rather than learning the subject).
New Way (Pairwise): The judge is shown two answers side-by-side and simply asked, "Which one is better?"
- Analogy: Instead of asking a food critic to rate a burger on a scale of 1 to 10 (which is hard to calibrate), you just ask, "Is Burger A better than Burger B?" This is much harder to cheat and gives a clearer signal of what "good" actually looks like.

The Results: Stronger and Smarter

The paper tested this method on two popular robot models (Llama and Qwen). Here is what they found:

The Guardian got tougher: The Defender models trained with AdvGame were much harder to trick. When tested against known "jailbreak" attacks (ways to bypass safety filters), they held their ground much better than models trained with old methods.
The Guardian stayed helpful: A common problem with safety training is that the robot becomes too cautious and refuses to answer harmless questions (like "How do I kill a computer process?"). AdvGame managed to keep the robot helpful and useful while still being safe.
The Trickster became a super-tool: The Attacker robot didn't just disappear after training. It became a powerful "Red Team" tool. This means the Attacker itself can now be used to test other robots to see if they are safe, acting as a professional stress-tester.

The Secret Sauce

The paper highlights three reasons why this worked so well:

Two Separate Robots: They didn't use one robot to play both roles (which can get confused). They used two distinct models, so the Attacker stays focused on attacking and the Defender on defending.
The "Better Than" Judge: Using the pairwise comparison (which is better?) prevented the robots from cheating the scoring system.
The "Moving Average" Trick: They used a technique called EMA (Exponential Moving Average). Imagine the Guardian is a student taking a test. Instead of panicking because they got one question wrong, they look at their performance over the last few weeks to see their true skill level. This keeps the training stable and prevents the robot from overreacting to a single bad example.

Summary

AdvGame is like a gym where a robot learns to dodge punches. Instead of a coach showing it a video of a punch and telling it how to dodge, the robot trains by sparring with a partner who is constantly inventing new punches in real-time. The result is a robot that is not only safer but also more useful, and a sparring partner that is so good at finding holes in defenses that it can be used to test any other robot in the future.

Technical Summary: Safety Alignment of LMs via Non-cooperative Games

Problem Statement

Ensuring the safety of Large Language Models (LMs) while preserving their utility remains a critical challenge. Current safety alignment approaches typically rely on sequential adversarial training: an Attacker LM generates adversarial prompts (jailbreaks), and a Defender LM is fine-tuned to resist them. This process is often applied iteratively in a "cat-and-mouse" fashion. The authors argue that this sequential alternation suffers from inefficiencies and potential instabilities inherent in alternating optimization. Furthermore, existing methods often struggle to maintain robustness against adaptive attacks, leading to non-trivial jailbreak success rates on standard benchmarks. Additionally, many approaches rely on point-wise reward signals from judge models, which are prone to reward hacking due to the difficulty of calibrating absolute safety scores and the subjective nature of safety violations.

Methodology: AdvGame

The paper introduces AdvGame, a novel paradigm that frames safety alignment as a non-zero-sum, non-cooperative game between two distinct agents: an Attacker LM ( $\rho$ ) and a Defender LM ( $\pi$ ). Unlike previous "self-play" approaches where a single model serves both roles (sharing parameters), AdvGame trains two separate models concurrently via online reinforcement learning.

Core Components

Non-Zero-Sum Formulation: The game is defined as non-zero-sum because the Attacker and Defender have distinct, not perfectly opposed objectives.
- Defender Objective: To produce safe, policy-compliant, and useful responses. For benign queries, it should be compliant; for harmful queries, it should provide a "deflection" (safe, semantically adjacent information) rather than a direct refusal or harmful output.
- Attacker Objective: To elicit failures (denial of service) or policy violations from the Defender. Crucially, the Attacker is rewarded for fooling the Defender into misclassifying the query type (e.g., making a benign query look harmful to trigger over-refusal, or a harmful query look benign to bypass safety). This prevents the Attacker from converging to degenerate solutions like generating gibberish, which would technically "win" in a zero-sum setup but fails to test robustness.
Preference-Based Optimization: Instead of using point-wise scalar rewards (which are difficult to calibrate), AdvGame utilizes pairwise preference signals.
- Judges: A judge LM evaluates pairs of responses (for the Defender) or pairs of attack prompts and their resulting responses (for the Attacker).
- Defender Judge: Selects the response that is more compliant (for benign seeds) or provides a better deflection (for harmful seeds).
- Attacker Judge: Selects the attack prompt that successfully tricks the Defender. The preference logic is swapped based on the seed type to ensure the Attacker learns to manipulate the Defender's classification rather than simply generating noise.
- Faithfulness: A separate judge ensures the Attacker's prompts remain faithful to the original seed intent, preventing the Attacker from changing the underlying meaning of the query.
Training Algorithm (AdvGame-DPO-MD):
- The method adapts Direct Preference Optimization (DPO) and Nash Mirror Descent (Nash-MD) concepts to this joint setting.
- Off-Policy Sampling: To improve training stability, the algorithm uses Exponential Moving Average (EMA) models ( $\pi^\gamma, \rho^\gamma$ ) to generate online data (rollouts) rather than using the current policy directly. This off-policy approach helps smooth updates and prevents the model from overfitting to recent changes, a common issue in pure on-policy RL.
- Loss Functions: The authors derive specific DPO-style loss functions for both the Attacker and Defender that optimize the Nash equilibrium of the game using the pairwise preference data.

Key Contributions

Joint Non-Cooperative Training: The paper proposes a framework where Attacker and Defender are trained simultaneously in a non-zero-sum game, moving away from sequential alternation or parameter-sharing self-play.
Preference-Based Supervision: The method replaces fragile point-wise reward signals with robust pairwise preference comparisons, mitigating reward hacking and calibration issues.
Off-Policy Stability: The integration of EMA-based off-policy sampling (AdvGame-DPO-MD) is shown to significantly improve training stability compared to on-policy methods like GRPO.
Dual-Utility Output: The framework produces not only a robust Defender but also a strong, general-purpose Attacker that converges into a capable red-teaming agent.

Experimental Results

The authors evaluated AdvGame on Llama3.1-8B and Qwen2.5-7B models against baselines including the original instruction-tuned models, uncensored versions, and the Self-RedTeam baseline.

Utility-Safety Tradeoff: AdvGame variants (particularly AdvGame-DPO-MD) successfully preserved the original model's utility (measured by MMLU, IFBench, AlpacaEval2, etc.) while significantly improving safety. In contrast, the Self-RedTeam baseline often resulted in noticeable drops in utility.
Safety Performance: On safety benchmarks like HarmBench, WildJailbreak (WJB), and WildGuardTest, AdvGame-DPO-MD achieved state-of-the-art performance, reducing Attack Success Rates (ASR) significantly more than baselines. For instance, on Qwen2.5-7B, AdvGame reduced HarmBench ASR to 4.7% compared to 16.8% for Self-RedTeam.
Robustness to Adaptive Attacks: The method demonstrated superior robustness against adaptive jailbreaking attacks (PAIR, TAP, GCG) compared to fixed prompt defenses and sequential training methods.
Attacker Capability: The trained Attacker LM achieved performance comparable to strong adaptive attack methods (e.g., achieving ~56% ASR on Qwen2.5-7B), validating its utility as a red-teaming tool.
Ablation Studies:
- DPO/IPO vs. GRPO: Preference-based methods (DPO/IPO) outperformed GRPO, which exhibited unstable training dynamics and lower safety scores.
- Pairwise vs. Point-wise: Pairwise judges yielded better safety and utility trade-offs, whereas point-wise judges led to reward hacking and performance drops.
- EMA vs. On-Policy: Using EMA for off-policy generation was critical for stability; removing it degraded safety metrics significantly.
- Training the Attacker: Actively training the Attacker (rather than keeping it fixed) was essential for generating diverse and challenging attacks, leading to a more robust Defender.

Significance and Claims

The paper claims that AdvGame shifts the Pareto frontier of safety and utility, yielding a Defender that is simultaneously more helpful and more resilient to adversarial attacks. By formalizing safety alignment as a non-cooperative game with preference-based rewards, the method addresses the inefficiencies of sequential training and the instability of point-wise reward optimization.

A key significance highlighted is the generation of a reusable red-teaming agent. The trained Attacker LM serves as a systematic tool for probing arbitrary target models, surfacing safety-relevant failure modes early in development. The authors emphasize that this approach supports a tighter feedback loop between discovery and remediation, improving the speed and coverage of safety assessments.

The work acknowledges limitations, noting that it focuses exclusively on natural-language attacks understandable by humans and does not cover non-readable attack sequences (e.g., GCG). However, it posits that future work in reward model design and extending to other attack modalities could further enhance the robustness of game-theoretic approaches to LM safety.