Technical Summary: Internalizing Safety Understanding in Large Reasoning Models via Verification

1. Problem Statement

Large Reasoning Models (LRMs) leverage explicit Chain-of-Thought (CoT) to enhance reasoning capabilities, but this coupling of reasoning with final answers amplifies the potential for real-world safety risks. Current alignment paradigms are predominantly answer-centric, relying on Supervised Fine-Tuning (SFT) on expert-curated safe trajectories or Reinforcement Learning (RL) with verifiable safety rewards. These methods optimize models to ensure final answers comply with external safety specifications.

However, the authors argue that this approach remains largely behavioral and superficial. It optimizes for compliance without shaping the model's intrinsic safety understanding. Consequently, ostensibly aligned models often lack the capacity to verify whether and why an output satisfies safety specifications. This gap manifests in two critical failures:

1. Vulnerability: Models can be hijacked by adversarial jailbreaks that induce compliance-inducing CoT, steering the model to label a prompt as "safe" while eliciting harmful answers.
2. Poor Generalization: Models merely imitate safe behaviors without learning the underlying concept of safety, leading to brittle defenses against out-of-domain (OOD) and reasoning-specific jailbreaks.

Empirical analysis reveals that standard alignment does not reliably yield verification skills; in some cases, aligned models perform worse than random baselines on binary safety verification tasks.

2. Methodology: Safety Internal (SInternal)

To address the decoupling of safety generation and safety understanding, the authors propose Safety Internal (SInternal), a framework designed to internalize safety specifications by training LRMs exclusively on safety verification tasks.

Core Philosophy

Instead of teaching models what to output (mimicking safe trajectories), SInternal teaches models how to evaluate their own outputs. The central hypothesis is that learning to verify induces a strong generalization for response safety, creating a more robust foundation for alignment than mere behavioral mimicry.

Framework Components

The methodology consists of two primary stages:

A. Safety Internalization via Verification Training

1. Self-Response Generation: For each safety-relevant prompt $x$, the policy model $\pi_\theta$ samples its own responses (including potentially unsafe ones) to ensure the model learns safety boundaries relevant to its actual output distribution.
2. Expert Verification Trajectory Construction: An expert model (e.g., Claude-4-Sonnet) evaluates the sampled responses against safety specifications $S$. It generates a verification trajectory $c_k$ comprising:
   • Critique Reasoning ($z_{ver,k}$): A detailed analysis of potential safety violations.
   • Binary Judgment ($v_k$): A final safety label (Safe/Unsafe).
3. Supervised Fine-Tuning (SFT): The policy model is trained via supervised learning on these expert verification trajectories. The objective is to maximize the likelihood of generating the verification reasoning and judgment conditioned on the prompt, the model's own response, and the safety specifications:
   $$L_{SInternal} = -E_{(x,y,c) \sim D_{ver}} [\log \pi_\theta(c | S, x, y)]$$
   This compels the model to internalize the logic of why an answer is unsafe.

B. Reinforcement Learning with Verifiable Rewards (RLVR)

Building upon the SInternal initialization, the framework further optimizes the model using Group Relative Policy Optimization (GRPO).

• Reward Design: The model generates a group of responses. An external verifier provides binary rewards based on whether the response is safe ($V_{safe}$) and whether it is a refusal ($V_{refuse}$).
• Objective: The model maximizes the expected group-relative reward, leveraging the internalized safety understanding to interpret reward signals more consistently and achieve higher alignment ceilings.

3. Key Contributions

1. Identification of the Verification Gap: The paper provides empirical evidence that answer-centric alignment fails to improve intrinsic safety verification capabilities, often leaving models unable to correctly identify unsafe outputs they generate.
2. SInternal Framework: A novel training paradigm that shifts the focus from generating safe answers to verifying them. By training exclusively on verification trajectories derived from the model's own generations, the method internalizes safety specifications as an intrinsic reasoning capability.
3. Superior Initialization for RL: The study demonstrates that SInternal serves as a superior initialization for Reinforcement Learning compared to standard SFT. Models initialized with SInternal achieve significantly higher robustness against adversarial attacks after RL optimization.
4. Mechanism Analysis: The authors show that SInternal triggers proactive verification during the reasoning process. The model spontaneously scrutinizes the safety of its potential behaviors before finalizing an answer, leading to deeper alignment.

4. Experimental Results

Experiments were conducted on three open-source LRMs (DeepSeek-R1-Distill-Qwen-7B/14B and DeepSeek-R1-Distill-Llama-8B) across safety, over-refusal, and general reasoning benchmarks.

• Robustness Against Jailbreaks: SInternal significantly outperforms baselines (SafeChain, STAR-1) on out-of-distribution (OOD) and reasoning-specific jailbreak benchmarks (e.g., WildJailbreak, Fortress, HCoT). Notably, SInternal combined with RL is the only method that effectively defends against HCoT (Hijacking CoT), a state-of-the-art jailbreak that exploits reasoning traces.
• Verification Capability: Models trained with SInternal show substantial gains in safety verification F1 scores (e.g., rising from ~48% to ~86% on DS-7B), whereas answer-centric baselines show little to no improvement.
• Generalization: The method demonstrates strong generalization to unseen jailbreaks, suggesting that learning the principles of safety verification is more effective than memorizing refusal patterns.
• Preservation of Utility: SInternal maintains competitive performance on general reasoning benchmarks (MATH, AIME) and avoids excessive over-refusal (measured by XSTest), preserving the balance between safety and utility.
• Data Efficiency: SInternal achieves higher safety performance with only 50% of the training data required by SFT baselines, indicating that internalizing reasoning logic is more data-efficient than mimicking outputs.

5. Significance and Claims

The paper claims a paradigm shift in AI safety alignment: moving from answer-centric alignment (mimicking safe behaviors) to intrinsic safety understanding (explicitly reasoning about safety principles).

• Foundational Robustness: The authors argue that internalizing safety understanding creates a more robust foundation for alignment than merely mimicking safe behaviors. This is evidenced by the model's ability to spontaneously trigger verification during reasoning and its superior resistance to adversarial CoT hijacking.
• Decoupling Generation and Verification: The work highlights that safe generation does not naturally lead to safety understanding. Therefore, explicit training on verification is necessary to bridge this gap.
• Future Direction: While the current framework focuses on post-generation verification, the authors suggest that extending this capability to dynamic self-verification during reasoning is a critical future direction. They emphasize that as models develop deeper intrinsic safety understanding, ensuring their internal verification logic remains aligned with human values is paramount for responsible governance.

The paper concludes that equipping LRMs with the capacity to scrutinize the safety of their own generated answers is a meaningful step toward achieving intrinsic safety, particularly in high-stakes environments where robustness is non-negotiable.