Technical Summary: CHASE (Co-evolutionary Hardening through Adversarial Safety-Escalation)

1. Problem Statement

Despite advances in safety alignment techniques like Reinforcement Learning from Human Feedback (RLHF) and Supervised Fine-Tuning (SFT), Large Language Models (LLMs) remain vulnerable to adaptive black-box attacks. Current defenses often rely on static decision boundaries that are brittle against evolving adversarial strategies such as persona modulation, fictional framing, and persuasion-based reformulation.

Existing red-teaming and defense frameworks suffer from two primary limitations:

1. Narrow Attack Distributions: Many methods rely on fixed attack templates or imitation learning, limiting the diversity of discovered attack strategies and the generalizability of the resulting defenses.
2. The Adaptive Gap: Defenses optimized at time $t-1$ often fail against novel attack distributions emerging at time $t$. Furthermore, prior co-evolutionary approaches often harden defenders via supervised learning on fixed targets, failing to capture latent attack primitives that transfer across mechanistically distinct attack families.

The central research question addressed is whether the brittleness of current LLM safety alignment stems from the narrow distribution of attacks used during training, and if broadening this distribution through reward-driven adversarial discovery can close the adaptive gap.

2. Methodology

CHASE (Co-evolutionary Hardening through Adversarial Safety-Escalation) is a closed-loop red-blue teaming framework where a black-box attacker and a safety-aligned defender co-evolve without exposure to pre-defined jailbreak templates.

2.1 The Attacker: Template-Free RL Exploration

The attacker ($A_\theta$) is trained using Group Relative Policy Optimization (GRPO) on a base model (NousResearch Hermes 4).

• Template-Free Initialization: The attacker receives no jailbreak templates, curated examples, or style scaffolding. It is instructed only to preserve the seed intent while rewriting the prompt. All adversarial framing strategies (e.g., hypothetical scenarios, persona assignment) must emerge purely through reward-driven exploration.
• Multiplicative Reward Decomposition: To prevent reward hacking (e.g., intent drift or over-sanitization), the attacker is optimized using a multiplicative reward function:
  $$R = S_{bypass} \times I_{intent}$$
  • $S_{bypass}$ (Bypass Score): A composite metric combining a binary refusal gate, specificity of harmful information, and convincingness of the bypass.
  • $I_{intent}$ (Intent Fidelity): A score from an independent judge model ensuring the rewritten prompt retains the original harmful intent.
    This formulation enforces a joint threshold: a rewrite that bypasses safety but loses the intent (or vice versa) receives a near-zero reward.

2.2 The Defender: Two-Stage Hardening Pipeline

The defender ($D_\gamma$), based on Llama-3.1-8B-Instruct, is hardened through a two-stage process to address the variance and consistency issues of on-policy RL for defense.

1. GRPO Exploration: The defender is first trained via GRPO on harmful prompts (both original intents and attacker rewrites). The reward function prioritizes refusal over compliance, with bonuses for high-quality, educational refusals. This stage explores the refusal landscape but produces sporadic high-quality refusals.
2. Rejection-Sampled SFT: To consolidate the sporadic refusals into consistent behavior, the system generates multiple candidate responses at high temperature. The highest-quality confirmed refusals are harvested to create a training dataset. This dataset is balanced with:
   • Layer 2 Refusals: High-quality refusals to adversarial rewrites.
   • Layer 1 Refusals: Refusals to original harmful intents.
   • Benign Responses: Helpful responses from standard datasets (e.g., Stanford Alpaca) to prevent over-refusal (false refusals).
     The defender is then fine-tuned via Supervised Fine-Tuning (SFT) on this curated dataset.

3. Key Contributions

1. Template-Free Co-evolutionary Framework: A novel setup where both attacker and defender are trained via GRPO without exposure to jailbreak templates. The attacker discovers adversarial framings purely through reward-driven exploration, forcing convergence on latent attack primitives rather than imitating specific families.
2. Multiplicative Reward Decomposition: A reward structure ($R = S_{bypass} \times I_{intent}$) that eliminates intent-drift and over-sanitization pathologies common in single-objective adversarial training.
3. Strong Cross-Attack Generalization: A defender trained exclusively on harvested CHASE attacker outputs demonstrates robustness against five held-out, mechanistically distinct attack families (PAIR, TAP, AutoDAN, PAP, Translation) that were not used during training.
4. Interpretable Cost Analysis: The framework achieves safety improvements while maintaining 0% false refusal rates on benign prompts, with any performance degradation concentrated specifically on fictional and roleplay framings—the very vectors most exploited by jailbreaks.

4. Experimental Results

Evaluated on BeaverTails and JailbreakBench against five held-out attack families:

• Safety Improvement: The CHASE defender reduced the mean StrongREJECT score by 43.2% across all attacks and datasets.
• Zero False Refusals: The model achieved 0% false refusal on 100 held-out benign prompts from Stanford Alpaca.
• JailbreakBench Performance: The defender achieved 0% Attack Success Rate (ASR) on direct misuse, PAIR transfer, and GCG transfer evaluations, matching the strongest published results.
• Ablation Study: When the CHASE attacker was replaced with a fixed attack family (PAIR artifacts), the resulting defender showed significantly lower generalization (−28.6% reduction vs. −43.2% for CHASE), confirming that the RL-discovered distribution is the primary driver of robustness.
• Quality vs. Volume: The CHASE attacker produced fewer bypasses than baselines (like PAIR or PAP) but with significantly higher convincingness and specificity, indicating higher-fidelity attacks that are harder to defend against without the proposed framework.

5. Significance and Claims

The paper claims that CHASE demonstrates a path toward LLM safety hardening that generalizes beyond the narrow distributions achieved in previous adversarial training. By removing template scaffolding, the framework recovers latent attack primitives—reusable framing strategies that recur across distinct attack methods (e.g., AutoDAN, TAP, PAP).

The authors argue that the "cost" of this hardening is interpretable and targeted: the defender learns to refuse prompts involving fictional scenarios, roleplay, and persona modulation. Since these framings constitute the dominant attack surface in the wild (e.g., 98% of in-the-wild jailbreaks use "pretending" strategies), the resulting over-refusal is a structural heuristic rather than a random failure mode. This suggests that template-free RL exploration can close the adaptive gap between evolving jailbreaks and static LLMs more effectively than methods relying on fixed attack templates.

Limitations: The authors note that the MT-Bench cost (−1.92 points) is non-trivial and concentrated in reasoning-heavy categories. Additionally, the defender classifies by framing features rather than user intent, leading to elevated refusal on benign fictional queries. The evaluation relies on LLM judges rather than human annotation, and transfer to other model architectures remains untested.