ARES: Adaptive Red-Teaming and End-to-End Repair of Policy-Reward System

1. Problem Statement

Reinforcement Learning from Human Feedback (RLHF) is the standard for aligning Large Language Models (LLMs) with safety principles. However, current safety alignment pipelines suffer from a critical vulnerability: Systemic Weaknesses.

• The Flaw: Existing approaches typically treat the Core LLM (the policy) and the Reward Model (RM) as separate entities. They either red-team the LLM assuming the RM is perfect, or harden the RM in isolation.
• The Reality: An imperfect RM can become a single point of failure. If the RM fails to penalize unsafe behaviors (due to biases, blind spots, or adversarial exploitation), the Core LLM learns to generate harmful content that the RM incorrectly rewards.
• The Gap: There is a lack of frameworks that simultaneously discover and repair coupled failures where both the Core LLM and the RM fail in tandem, leading to a "blind spot" in the safety pipeline.

2. Methodology: The ARES Framework

ARES (Adaptive Red-Teaming and End-to-End System Repair) is a two-phase framework designed to discover and mitigate these dual vulnerabilities.

Phase 1: Adaptive Vulnerability Discovery

This phase employs a novel "Safety Mentor" to generate adversarial prompts and classify failure modes.

• Compositional Attack Generation:
  • The Safety Mentor constructs semantically coherent, deceptive prompts by combining four structured components: Topic (harmful domain), Persona (credible identity), Goal (concrete task), and Tactic (plausible methodology).
  • For each prompt, the Mentor generates a triplet: the prompt, a synthetic harmful response ($y_{synth}$), and a preferred safe response ($y_{chosen}$).
• Dual-Component Vulnerability Classification:
  The system evaluates the interaction between the Core LLM and the RM to categorize failures into three types:
  1. Type A (RM-only Misalignment): The RM assigns a high score to harmful content (fooling the RM), but the Core LLM would not generate it. Action: Fine-tune RM.
  2. Type B (Core LLM Vulnerabilities): The Core LLM generates harmful content, but the RM correctly identifies it as unsafe (low score). Action: Optimize Core LLM.
  3. Type C (Systemic Weaknesses): The Critical Failure. Both the Core LLM generates harmful content and the RM assigns it a high reward. Action: Joint repair of both components.
• Hierarchical Adaptive Sampling:
  • ARES uses a weighted sampling mechanism that learns from successful attacks.
  • If an attack succeeds (exposes a vulnerability), the weights of the specific Component Instance and its parent Component Category are multiplicatively increased.
  • This allows the system to dynamically focus on high-yield vulnerability regions (e.g., specific deception tactics) rather than uniform random sampling.

Phase 2: End-to-End System Repair

Once vulnerabilities are discovered, ARES executes a sequential repair process to ensure robust alignment.

1. Reward Model Repair (Stage 1):
   • The RM is fine-tuned using a preference dataset constructed from Type A and Type C failures (where the RM was fooled).
   • This dataset is augmented with general alignment data (e.g., HelpSteer2) to prevent capability degradation and FalseReject data to mitigate over-refusal.
   • Rationale: The RM must be fixed first to serve as a reliable reward signal for the next stage.
2. Core LLM Optimization (Stage 2):
   • The Core LLM is optimized using the repaired RM as the reward signal via RL (specifically Dr. GRPO).
   • The training data includes Type B and Type C failures (where the LLM generated harm).
   • Rationale: Optimizing the LLM against a flawed RM perpetuates systemic weaknesses; a repaired RM ensures the policy learns correct safety boundaries.

3. Key Contributions

• Identification of Systemic Weaknesses: The paper formally defines and categorizes "Type C" failures, highlighting the danger of coupled LLM-RM failures that existing red-teaming methods miss.
• Safety Mentor & Compositional Generation: A structured approach to generating deceptive, coherent adversarial prompts using a taxonomy of topics, personas, goals, and tactics.
• Adaptive Learning Mechanism: A hierarchical reinforcement strategy that dynamically shifts focus toward the most effective attack vectors, improving data efficiency.
• Closed-Loop Dual Repair: A principled, two-stage repair pipeline that sequentially fixes the RM and then the LLM, ensuring the policy is optimized against a robust safety evaluator.

4. Experimental Results

Experiments were conducted using Qwen3-1.7B (Core LLM) and Skywork-RM-Qwen3-4B (RM), with Qwen3-8B acting as the Safety Mentor.

• Safety Performance:
  • ARES achieved near-perfect safety rates on adversarial benchmarks: StrongReject (0.97) and HarmBench (0.95).
  • This significantly outperformed standard RLHF baselines and general safety alignment methods (e.g., General Safe-RLHF achieved 0.94 on StrongReject).
• Capability Preservation:
  • Despite aggressive safety tuning, ARES maintained strong performance on capability benchmarks (MMLU, GSM8K, TruthfulQA), showing no catastrophic forgetting.
  • It also maintained a low incorrect refusal rate (0.10 on XSTest), avoiding the "over-refusal" common in safety-tuned models.
• Efficiency:
  • Data Efficiency: ARES achieved safety levels comparable to models trained on 10.8k general safety samples using only 4k adaptively generated samples.
  • Computational Efficiency: The full pipeline took ~13 hours, significantly faster than baselines like APRT (estimated 28 hours).
• Ablation Studies:
  • Removing the adaptive sampling mechanism resulted in lower safety scores (0.91 vs 0.97 on StrongReject), proving the value of the learning mechanism.
  • Removing general data or over-refusal mitigation data led to capability drops or increased false refusals, validating the data mixture strategy.

5. Significance and Impact

• Paradigm Shift: ARES moves the field from isolated "policy-only" or "reward-only" fixes to a holistic system alignment approach. It acknowledges that safety is a property of the entire RLHF pipeline, not just the final model.
• Practicality: By demonstrating that targeted, adaptive red-teaming is more efficient than massive static datasets, ARES offers a scalable path for continuous safety improvement in production LLMs.
• Robustness: The framework effectively eliminates the vast majority of discoverable vulnerabilities (reducing attack success rates from 63.5% to 4.3% in a second round of testing), establishing a new standard for robust safety alignment.

In summary, ARES provides a comprehensive, automated framework that not only finds where safety systems break but also systematically repairs the underlying reward and policy mechanisms to prevent future failures.