🤖 machine learning

Constitutional On-Policy Safe Distillation

This paper introduces Constitutional On-Policy Safe Distillation (COPSD), a method that addresses the severe collapse and reduced expressiveness observed in prior safety distillation approaches by calibrating the teacher model via Cross-SFT and employing constitution-conditioned on-policy distillation to achieve a superior safety-helpfulness trade-off across 12 benchmarks.

Original authors: Ming Wen, Yuxuan Liu, Kun Yang, Yunhao Feng, Zhuoer Xu, Yuhao Sun, Shiwen Cui, Xiang Zheng, Xingjun Ma, Yu-Gang Jiang

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ming Wen, Yuxuan Liu, Kun Yang, Yunhao Feng, Zhuoer Xu, Yuhao Sun, Shiwen Cui, Xiang Zheng, Xingjun Ma, Yu-Gang Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching an AI to be Safe Without Being Boring

Imagine you are training a very smart, chatty robot (the "Student") to be helpful but also safe. You want it to answer questions about the world without ever saying something dangerous or offensive.

The researchers found that a popular method for teaching robots safety was actually making them stupidly cautious. Instead of giving thoughtful, nuanced answers, the robots started giving short, robotic, "I can't help with that" responses to almost everything. They became safe, but they also became useless.

This paper introduces a new method called COPSD that fixes this. It teaches the robot to be safe while still keeping its personality, creativity, and ability to explain things.

The Problem: The "Over-Cautious Teacher" Trap

To understand the problem, imagine a classroom scenario:

The Old Way (OPSD): You have a "Teacher" robot that knows the safety rules (the "Constitution"). The "Student" robot tries to copy the Teacher.
The Glitch: When the Teacher is told to be safe, it gets scared. It starts giving very short, boring answers like, "That is dangerous. Do not do it." It stops explaining why or offering safe alternatives.
The Collapse: The Student robot looks at the Teacher and thinks, "Oh, to be safe, I just need to be short and say 'No'." The Student copies this behavior perfectly.
- Result: The robot becomes a "safety tax." It is so afraid of making a mistake that it refuses to answer helpful questions, or it gives one-sentence answers that aren't actually helpful.

The Paper's Discovery:
The researchers realized this wasn't because the robot was "cheating" by copying answers (like in math problems). Instead, it was a geometric problem.

Imagine safety and "expressiveness" (being chatty and helpful) are two directions on a map.
In a perfect world, you could move "North" (Safety) without moving "West" (Helpfulness).
But in this robot's brain, the map is tilted. When you push the robot hard toward "Safety," the tilt forces it to slide backward into "Helplessness." The pressure to be safe accidentally crushed its ability to be expressive.

The Solution: COPSD (The Two-Step Fix)

The authors propose a two-step training process to untangle this tilted map.

Step 1: The "Cross-SFT Cold-Start" (Calibrating the Teacher)

Before the Student starts learning, the Teacher needs to be fixed first.

The Analogy: Imagine the Teacher is a strict parent who only knows how to say "No." The researchers give the Teacher a special training course. They show the Teacher examples of how to say "No" gracefully—explaining why something is bad and offering a safe alternative, all while keeping the tone friendly and detailed.
The Result: The Teacher learns to be safe without being a short, boring robot. It learns that safety and being helpful can coexist.

Step 2: On-Policy Distillation (The Student Learns from the Fixed Teacher)

Now, the Student robot starts learning from this newly calibrated Teacher.

The Analogy: The Student watches the Teacher give those perfect, safe, yet detailed answers. Because the Teacher isn't just saying "No" anymore, the Student learns that it can be safe and detailed.
The Magic: The Student doesn't just copy the words; it learns the pattern of being safe without losing its voice.

What Happened When They Tested It?

The researchers tested this new method (COPSD) against other popular methods on 12 different tests.

Safety vs. Helpfulness:
- Other Methods: When they made the robots safer, the robots became much less helpful (the "Safety Tax" went up).
- COPSD: The robots became safer and stayed helpful. They achieved a "Pareto Improvement," meaning they got better at safety without getting worse at anything else. In fact, they were safer than a much larger, more expensive robot model.
General Intelligence:
- Other Methods: When trying to make robots safe, their ability to do math or solve logic puzzles often dropped significantly.
- COPSD: The robots kept their math and reasoning skills almost perfectly intact. The "Safety Tax" on their brainpower was almost zero.
Breaking the Ceiling:
- Usually, a student can't be better than the teacher. But because the COPSD Teacher was so well-calibrated, the Student actually learned to be better than the Teacher at balancing safety and helpfulness.

Summary in One Sentence

The paper shows that trying to teach AI safety often accidentally makes it boring and unhelpful because the training method pushes safety and helpfulness in the wrong direction; their new method, COPSD, first fixes the teacher to be "safely expressive," allowing the student to learn safety without losing its personality or smarts.

Technical Summary: Constitutional On-Policy Safe Distillation (COPSD)

1. Problem Statement

On-Policy Self-Distillation (OPSD) has emerged as an efficient post-training paradigm for aligning foundation models, utilizing a teacher augmented with privileged information to provide dense token-level supervision. While effective in verifiable reasoning tasks, recent studies indicate that OPSD suffers from severe "distributional collapse" in these domains, where students exploit superficial correlations to minimize divergence penalties.

This paper identifies that safety alignment presents a distinct and critical failure mode for OPSD. Unlike reasoning tasks where collapse stems from shortcut copying of explicit target answers, safety alignment relies on high-level constitutions (principles) rather than explicit answers. The authors' pilot study reveals that safety OPSD collapses due to a "geometric leakage" effect:

Teacher Contraction: Conditioning the teacher on safety constitutions forces the teacher distribution to contract into a manifold of short, overly conservative, and low-entropy responses.
Reverse KL Amplification: The standard Reverse KL divergence used in distillation amplifies this contraction, penalizing any token outside the teacher's narrow support.
Geometric Leakage: In a non-orthogonal semantic space where "safety" and "expressiveness" are coupled, the gradient pressure to avoid unsafe regions leaks into the expressiveness dimension. This results in a passive contraction of response length, diversity, and helpfulness, even without the student copying privileged information.

The core challenge is the "safety tax": achieving robust safety alignment often necessitates a drastic reduction in general reasoning capabilities and helpfulness, a trade-off existing methods (like GRPO or standard OPD) fail to navigate effectively.

2. Methodology: Constitutional On-Policy Safe Distillation (COPSD)

To address the geometric entanglement between safety and expressiveness, the authors propose COPSD, a two-stage framework designed to recalibrate the teacher before distillation.

Phase I: Cross-SFT Cold-Start

The first stage focuses on constructing a constitution-conditioned teacher that preserves the base model's generative footprint while adhering to safety principles.

Data Construction: The training corpus is a mixture of:
1. Original Trajectories: Original queries and responses from the base model, augmented with the safety constitution $c_{safe}$ . This anchors the teacher to the model's native expressiveness.
2. Rewritten Safety Trajectories: High-quality safety responses generated by frontier models, filtered by LLM judges, rewritten by the base model, and resampled to match the original response-length distribution.
Objective: This "Cross-SFT" process trains the teacher to internalize safety boundaries without collapsing into uniformly short or over-cautious outputs, effectively decoupling the safety manifold from the expressiveness manifold.

Phase II: Constitution-Conditioned On-Policy Distillation

The recalibrated teacher is then integrated into an on-policy distillation pipeline.

Teacher Role: The teacher generates dense token-level supervision signals conditioned on both the query $x$ and the safety constitution $c_{safe}$ .
Distillation Signal: The student policy is optimized to minimize the token-level Reverse KL divergence against the constitution-conditioned teacher.
Advantage Estimation: The framework supports two variants:
1. Standard Token-Level: Relies entirely on dense distillation signals to compute group-relative advantages.
2. Hybrid Outcome-Weighted: Combines the dense token-level signal with an external sequence-level outcome reward (e.g., from a reward model) to enforce global safety preferences while refining local credit assignment.

3. Key Contributions

Diagnosis of Safety-Specific Collapse: The paper formalizes the failure of safety OPSD as "geometric leakage" under safety boundaries in a non-orthogonal semantic space. It demonstrates that this collapse is distinct from the "shortcut copying" observed in verifiable reasoning tasks; it arises because safety pressure spills over into the expressiveness dimension due to the coupling of safety and linguistic priors.
COPSD Framework: The authors propose a two-stage solution that uses Cross-SFT Cold-Start to calibrate the teacher distribution prior to distillation. This approach mitigates the safety-expressiveness entanglement, allowing the teacher to provide dense supervision without forcing the student into a low-diversity, over-cautious regime.
Empirical Validation: Extensive experiments on 12 benchmarks (covering safety, situational safety, and general reasoning) demonstrate that COPSD achieves a superior safety-helpfulness trade-off compared to state-of-the-art baselines (GRPO, OPD, Safe-RLHF-V).

4. Experimental Results

Safety vs. Helpfulness: On benchmarks like BeaverTails-V and SPA-VL, COPSD variants achieve significantly higher win rates against the base model in both safety and helpfulness compared to baselines. For instance, while GRPO improves safety, it often incurs a severe drop in helpfulness (the "safety tax"). COPSD-Hybrid maintains robust effectiveness while reaching higher safety frontiers.
Preservation of General Capabilities: COPSD substantially reduces the "alignment tax" on general reasoning. On MathVista, COPSD limits reasoning loss to ~2.1%, whereas GRPO suffers a catastrophic 17.5% drop. Simple VQA tasks remain largely unaffected across all methods, but complex reasoning tasks benefit significantly from COPSD's decoupling mechanism.
Breaking the Teacher Ceiling: Unlike standard OPD, where the student is limited by the teacher's inherent capabilities, COPSD students (trained on the Cross-SFT teacher) significantly outperform both their specific teacher and standard OPD teachers in safety and effectiveness, indicating adaptive policy learning.
Training Dynamics: Analysis of training dynamics shows that while GRPO suffers from continuous entropy collapse and standard OPSD triggers a severe decline in response length, COPSD stabilizes quickly, maintaining higher entropy and stable response lengths throughout training.

5. Significance and Claims

The paper claims that safety alignment requires a fundamentally different approach than verifiable reasoning alignment. The significance of COPSD lies in its ability to:

Decouple Competing Objectives: By using Cross-SFT to calibrate the teacher, the method breaks the geometric coupling between safety constraints and expressive behavior, preventing the "leakage" of safety pressure into reduced helpfulness.
Enable Dense Supervision without Collapse: It demonstrates that dense token-level supervision is viable for safety alignment if the teacher distribution is properly anchored to the base model's generative style, avoiding the distributional collapse that plagues current methods.
Achieve Pareto Improvement: The results suggest that it is possible to simultaneously improve safety and preserve (or even enhance) general reasoning capabilities, challenging the notion that safety alignment must inherently degrade model utility.

The authors conclude that calibrating teacher distributions is a necessary step for stable and effective safe distillation, offering a pathway to align models with high-level constitutions without sacrificing their core generative abilities.