Technical Summary: Addressing Over-Refusal in LLMs with Competing Rewards

Problem Statement

Safety training on Large Language Models (LLMs) frequently induces over-refusal, a phenomenon where models become excessively cautious, refusing harmless or ambiguous prompts while attempting to mitigate harmful outputs. While Reinforcement Learning (RL) approaches that encourage explicit reasoning have been proposed to help models distinguish between harmful and benign intents, current methods often fail to fully resolve this trade-off.

The authors identify a critical limitation in existing safety-trained models: reasoning often acts as a "rubber stamp" for a predetermined refusal rather than a mechanism for genuine ambiguity resolution. These models exhibit two brittle behaviors:

1. They avoid unsafe interpretations entirely and default to refusal.
2. They explore unsafe possibilities but fail to recover to a safe final answer, leading to unsafe compliance.

Unlike mathematical reasoning, where models can explore incorrect paths and self-correct, safety reasoning in current models lacks the capability to explore potentially harmful interpretations in a controlled manner and then "flip back" to a safe response.

Methodology: SEAR and Competing Rewards

The paper proposes SEAR (Safety Exploration through Adversarial Reasoning), a framework that trains a single model to engage in harmful reasoning as an exploratory signal while ensuring the final output remains safe. This is achieved through an adversarial optimization problem cast within a single chain-of-thought.

Core Mechanism: Competing Rewards

The authors introduce a training objective that assigns separate, competing rewards to different segments of the model's generation trace:

• Reasoning Segment ($z$): Rewarded for harmfulness ($r_h$). The model is incentivized to explore unsafe interpretations and reasoning strategies.
• Answer Segment ($y$): Rewarded for safety ($r_s$). The model is incentivized to produce a safe final response conditioned on the preceding (potentially harmful) reasoning.

The objective function for harmful prompts ($D_h$) is:
$$\max_{\pi} \mathbb{E}_{x \sim D_h, (z,y) \sim \pi} [r_h(z) + r_s(y)]$$
For harmless prompts ($D_i$), the model is optimized for helpfulness ($r_i$). The training uses a KL-divergence regularizer to keep the policy close to a base model.

Key Technical Insight: Process Rewards

The paper demonstrates that standard dense outcome-based rewards (assigning a single scalar reward to the entire trajectory) are insufficient for this task. They lead to optimization instability where the model either fails to explore harmful reasoning or degrades performance on harmless tasks due to a mismatch in convergence rates between the competing objectives.

To solve this, the authors implement process-based rewards. Instead of a single reward for the whole rollout, rewards are assigned token-level to specific segments:

• Tokens in the reasoning segment receive the reasoning reward ($r_h$).
• Tokens in the answer segment receive the answer reward ($r_s$).

This "process reward" formulation allows for precise credit assignment, enabling the model to learn the coupled behavior of exploring harmful paths and recovering to a safe answer within a single trajectory.

Key Contributions

1. Reframing Safety Reasoning: The authors propose that unsafe reasoning should not be preemptively blocked but rather encouraged as a useful exploratory signal to resolve ambiguity, provided the model can reliably recover to a safe answer.
2. Competing Reward Framework: They introduce a method to train a single model with opposing objectives (harmful exploration vs. safe output) within a single chain-of-thought, moving beyond the "rubber stamp" behavior of current models.
3. Process Rewards for Stability: The paper identifies that dense outcome-level rewards fail to stabilize competing objectives and demonstrates that segment-level process rewards are crucial for successful optimization.
4. SEAR-1.5B Model: They train a 1.5B parameter model (SEAR-1.5B) that explicitly learns to "flip back" from harmful reasoning to safe answers.

Results

The authors evaluate SEAR-1.5B against various baselines, including standard outcome-based RL, dense outcome-based RL, SFT baselines, and frontier open-source models (e.g., TARS, Qwen3).

• Safety-Refusal Trade-off: SEAR-1.5B achieves the best performance on the Pareto frontier of safety and compliance (quadratic mean score of 0.930), outperforming TARS-7B (0.909) and other RL baselines. It demonstrates significantly less over-refusal on ambiguous prompts (XSTest, Fortress, False-Reject) while maintaining high defense success rates against jailbreak attacks (HarmBench).
• Exploration and Recovery: Analysis of reasoning traces shows that SEAR generates a higher frequency of harmful reasoning compared to other models but successfully recovers to safe answers. In contrast, other models either avoid harmful reasoning entirely or fail to recover when they do explore it.
• Robustness to Pre-fill Attacks: SEAR-1.5B exhibits superior robustness against attacks that pre-fill the model's context with harmful reasoning (e.g., using an abliterated model to generate the start of a harmful chain). SEAR maintains higher answer safety scores than outcome-based RL models and larger Qwen3 variants, demonstrating its ability to "flip back" even when forced into a harmful reasoning state.
• Generalization: The model maintains strong out-of-distribution (OOD) generalization on math and reasoning benchmarks (GSM8K, MATH-500, GPQA), comparable to or better than outcome-based baselines.

Significance and Claims

The paper claims that the safety-refusal trade-off is not merely a decision of whether to refuse, but a capability of how to resolve ambiguity. By learning to reason through unsafe possibilities without allowing those possibilities to dictate the final response, models can remain helpful on benign prompts without compromising safety.

The authors emphasize that their approach:

• Mitigates over-refusal by enabling models to distinguish harmful from harmless intent through genuine exploration rather than preemptive suppression.
• Provides a defense against reasoning-directed attacks (like pre-filling) by training the model to recover from harmful contexts.
• Establishes that process-based rewards are a necessary component for optimizing competing objectives in RL for safety, a finding that may generalize to other multi-objective RL tasks.

The work is presented as a research contribution to safety alignment, noting that the model and methodology are designed for research purposes to improve the robustness and helpfulness of LLMs.