🤖 AI

Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking

The paper proposes TRACE, a turn-aware credit assignment framework for reinforcement learning-based multi-turn jailbreaking that addresses the limitations of coarse trajectory-level supervision by estimating turn-level contributions and assigning targeted penalties, thereby significantly improving attack success rates and enabling more effective multi-turn defense alignment.

Original authors: Zhida He, Xiaoyu Wen, Han Qi, Ziyuan Zhou, Peng Yu, Xingcheng Xu, Dongrui Liu, Xia Hu, Chaochao Lu, Qiaosheng Zhang

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhida He, Xiaoyu Wen, Han Qi, Ziyuan Zhou, Peng Yu, Xingcheng Xu, Dongrui Liu, Xia Hu, Chaochao Lu, Qiaosheng Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Long Con" Against AI

Imagine you are trying to trick a very strict librarian (the AI) into giving you a book that is banned from the library.

The Old Way (Single Turn): You walk up and ask, "Give me the banned book!" The librarian immediately says, "No, that's against the rules." You fail.
The Multi-Turn Way: You try a "long con." You start by asking about the history of libraries, then ask about the author of the banned book, then ask about the chemical composition of the ink used in the book. Slowly, over many conversations, you build a context where the librarian forgets the rules and accidentally hands you the banned book.

This paper is about how to make that "long con" much more effective. The researchers found that current methods are like a coach who gives a player a gold medal for the entire game just because they won, even if the player spent 90% of the game standing still or making mistakes.

The Problem: The "Blame Game" is Broken

The researchers discovered that in these multi-turn conversations, not every turn is equally important.

The "Redundant" Turns: Sometimes, the AI asks a question, the librarian answers, and the AI asks the exact same thing again. This doesn't help the attack; it's just wasted time. But old methods gave this useless turn a "gold star" just because the attack eventually succeeded.
The "Critical" Turns: Sometimes, the AI asks a very specific, clever question that tricks the librarian into lowering their guard. This is the real key to success.
The "Phase" Problem: Asking for the banned book in the first sentence is too aggressive and gets you kicked out. But asking for it in the 10th sentence, after building trust, might work. Old methods didn't understand that timing matters.

The Result: The AI attackers were learning the wrong lessons. They thought, "Oh, I should just talk a lot and repeat myself," because that's what the "gold star" (reward) told them to do. They weren't learning how to be clever.

The Solution: TRACE (The Smart Coach)

The authors propose a new system called TRACE. Think of TRACE as a super-smart coach who watches the game tape and gives credit (or blame) to specific moments rather than the whole game.

How TRACE Works:

1. For Winning Games (Successful Attacks): The "What If?" Test
When the AI successfully tricks the librarian, TRACE looks at the conversation and asks: "What if we removed this specific turn?"

If removing a turn makes the attack fail, TRACE says: "Great job! That turn was critical. You get a huge reward."
If removing a turn doesn't change the outcome, TRACE says: "You were just wasting time. You get a small reward."
Analogy: It's like a detective realizing that the suspect's alibi only worked because of one specific lie. The detective focuses on that lie, not the whole story.

2. For Losing Games (Failed Attacks): The "Wrong Turn" Penalty
When the AI fails, TRACE doesn't just say "Bad job." It looks at why it failed.

Did the AI get too aggressive too early? (Too much "harmfulness" too soon).
Did the AI drift off-topic and forget the original goal? (Loss of "relevance").
TRACE assigns a penalty to those specific bad turns, teaching the AI: "Don't be too aggressive at the start, and don't forget what you're trying to do."

3. The "Refusal" Detector
TRACE also pays attention when the librarian says "No." If the AI asks something and gets a refusal, TRACE marks that turn as a "bad move" for that specific librarian, teaching the AI to try a different approach next time.

The Results: Smarter, Faster, Stronger

The researchers tested TRACE against many other methods on different types of "librarians" (AI models).

Better Success Rate: TRACE was about 25% more successful at tricking the AI than the previous best methods.
More Efficient: It didn't need to talk as much. It learned to skip the "redundant" turns and get straight to the "critical" ones.
Better at Adapting: Because TRACE learned why a turn worked (the specific strategy), it could use that same strategy on different AI models, not just the one it practiced on.

The Twist: Using the Attack to Build a Better Defense

The most interesting part of the paper is that the researchers didn't just stop at breaking the AI. They used the "credit scores" from TRACE to fix the AI.

The Insight: TRACE identified which turns were "latent risks"—moments where the conversation looked harmless but was actually setting a trap.
The Fix: They taught the AI (the librarian) to recognize these "trap-setting" moments. Instead of waiting until the very end to say "No," the librarian learned to say, "I can answer this, but I need to be careful not to give you the tools to misuse this information," earlier in the conversation.
The Result: The AI became safer without becoming useless. It learned to draw a line in the sand earlier, protecting itself from the "long con" while still being helpful to honest users.

Summary

In short, this paper says: "Stop treating every step of a conversation the same."

By teaching AI attackers to recognize which specific steps actually matter (and which are just noise), they became much better at breaking safety rules. But by using that same knowledge, they also taught AI defenders how to spot the danger earlier, making the whole system safer and smarter.

Technical Summary: Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking

1. Problem Statement

The deployment of Large Language Models (LLMs) in multi-turn dialogues has introduced a significant vulnerability: adversarial actors can distribute harmful intent across multiple seemingly benign turns, gradually accumulating context to bypass safety mechanisms. While existing training-based multi-turn jailbreak methods utilize reinforcement learning (RL) to learn long-horizon attack strategies, they suffer from a fundamental credit assignment problem.

Current RL-based approaches typically broadcast a single trajectory-level outcome signal (e.g., success or failure of the final response) uniformly to every turn in the dialogue. This coarse supervision leads to two critical issues:

Over-rewarding Redundancy: In successful trajectories, turns that contribute little to the final jailbreak receive the same positive reward as critical turns, reinforcing redundant behavior.
Under-crediting Useful Intermediates: In failed trajectories, turns that were strategically useful but ultimately failed due to other factors receive uniform negative signals, while turns that actively suppressed the attack (safety-critical) are not distinguished from neutral failures.

The authors identify that turn-level contributions are non-uniform (only a few turns drive success), phase-dependent (effectiveness depends on the dialogue stage), and target-specific (safety boundaries vary by model).

2. Methodology: TRACE

To address these limitations, the authors propose TRACE (TuRn-level Assignment for CrEdit), a framework that replaces uniform trajectory-level rewards with turn-aware credit assignment within a Group Relative Policy Optimization (GRPO) setting.

Core Components

TRACE modifies the advantage calculation in GRPO by introducing a turn-aware multiplier ( $m_{i,t}$ ) and a local process penalty ( $\hat{A}^p_{i,t}$ ):
$\hat{A}_{i,t} = m_{i,t} \hat{A}^o_i + \hat{A}^p_{i,t}$

A. Success-Side Credit Assignment (Leave-One-Turn-Out)

For successful trajectories ( $\tau_i \in S^+$ ), TRACE estimates the contribution of each intermediate turn using leave-one-turn-out semantic masking:

For each non-final turn $t$ , the interaction pair $(x_t, y_t)$ is removed from the history.
The final target response is resampled under this masked history.
The change in harmfulness between the original and masked trajectories defines the raw credit.
This credit is normalized and converted into a multiplier ( $m^+_{i,t}$ ), distinguishing attack-critical turns (necessary for success) from redundant turns.

B. Failure-Side Turn-Aware Penalty

For failed trajectories ( $\tau_i \in S^-$ ), semantic masking is less effective as the outcome is already safe. Instead, TRACE assigns penalties based on deviation from successful priors:

Harmfulness Penalty: Compares the harmfulness label of the current prompt against a target-specific success prior distribution ( $q_t$ ). It penalizes prompts that are atypical for the current phase (e.g., premature unsafe prompts or harmfulness drift).
Relevance Penalty: Measures the semantic similarity between the current prompt and the original harmful seed. It penalizes turns where the dialogue drifts too far from the original intent.
These penalties are combined into a failure-side multiplier ( $m^-_{i,t}$ ) that down-weights turns inconsistent with successful attack patterns.

C. Refusal-Aware Local Process Penalty

To further refine the learning signal, TRACE introduces a local penalty for turns that trigger a refusal from the target model. This penalty is phase-aware, applying stronger penalties to early refusals that indicate a failed strategy, encouraging the attacker to adapt more quickly.

Defense Alignment

The authors demonstrate that the credit signals extracted during the attack phase can be repurposed for multi-turn defense. By identifying "latent-risk" turns (benign intermediate steps that support the final attack) and "direct-harm" turns, TRACE constructs turn-level preference data. This data is used to align the defender via Direct Preference Optimization (DPO), enabling early risk intervention without over-refusal.

3. Key Contributions

Characterization of the Credit Assignment Problem: The paper empirically demonstrates that turn-level contributions in multi-turn jailbreaking are non-uniform, phase-dependent, and target-specific, invalidating the assumption of uniform trajectory-level rewards.
TRACE Framework: A novel credit assignment mechanism that utilizes leave-one-turn-out masking for successful trajectories and phase-aware deviation penalties for failed trajectories.
Defense Application: A method to reuse attack-side credit signals to construct turn-level preference data for defense, distinguishing between latent risks and direct harms to improve the safety-utility trade-off.
Comprehensive Evaluation: Extensive experiments across open-source and closed-source targets, demonstrating improvements in effectiveness, transferability, and efficiency.

4. Experimental Results

The authors evaluated TRACE against single-turn baselines (e.g., PAIR, AutoDAN), multi-turn workflow methods (e.g., X-Teaming, Crescendo), and training-based RL baselines (e.g., Siren, TROJail) on benchmarks including HarmBench, JailbreakBench, and WildJailBreak.

Effectiveness: TRACE achieves the strongest overall performance. Compared to the strongest RL baseline (TROJail), TRACE (mix) yields a ~25% relative improvement in Attack Success Rate (ASR), reaching an average ASR of 87.10% compared to TROJail's 69.97%. It significantly outperforms workflow-based methods even when using a smaller attacker model (Qwen2.5-3B) against targets like GPT-4o.
Transferability: While single-target training leads to target-specific policies, TRACE (mix)—jointly trained on two strong safety targets (gpt-oss-20b and Llama3.1-8B-IT)—demonstrates strong cross-target transferability. It achieves high ASR on unseen closed-source models, including 96.7% on GPT-4o and 93.1% on Gemini-2.5-Pro.
Efficiency: TRACE achieves high ASR with fewer turns and target calls. It reaches over 80% ASR on gpt-oss-20b within four turns, whereas workflow baselines remain significantly lower even with larger turn limits.
Defense Alignment: When used for defense, TRACE-based alignment reduces ASR for both multi-turn and single-turn attacks more effectively than prior methods (SafeMT, MUSE-D) while maintaining higher general capability scores (GPQA, GSM8K, MMLU).

5. Significance and Claims

The paper claims that TRACE represents a significant advancement in the understanding and execution of multi-turn adversarial attacks. By shifting from coarse trajectory-level supervision to fine-grained turn-level credit assignment, the authors argue that:

Attackers can learn more strategic, less redundant, and more efficient jailbreak policies.
The credit assignment problem is a primary bottleneck in current RL-based jailbreaking, and addressing it yields substantial performance gains.
The signals derived from attack-side credit assignment are valuable not just for breaking models, but for defending them. By identifying latent-risk turns, defenses can intervene earlier in the dialogue, offering a better balance between safety and utility (helpfulness) compared to methods that only react to explicit harmful outputs.

The authors conclude that their work highlights the necessity of turn-aware optimization in multi-turn LLM interactions and provides a framework that is both a potent red-teaming tool and a foundation for more robust defense mechanisms.