🤖 machine learning

DUET: Dual-Teacher On-Policy Distillation via Same-Weight Disagreement for Prohibition Compliance

The paper proposes DUET, a token-selective on-policy distillation method that leverages the per-token disagreement between two identical-weight teachers (one with and one without prohibition context) to generate a clean supervision signal for training models to comply with dynamic, request-specific prohibitions while preserving general utility.

Original authors: Zihan Li, Feifei Li, Wenhui Que

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zihan Li, Feifei Li, Wenhui Que

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Large language models are the engines behind many modern digital assistants, capable of generating human-like text, solving problems, and answering questions. However, in the real world, these models often operate under strict, shifting rules that are not hard-coded into their brains but are instead injected into their instructions at the moment they are used. A company might tell a model to never reveal its internal tools, while a hospital might demand it never mention patient names. These rules change depending on who is asking and what the situation is. The challenge for developers is to teach the model to obey these specific, temporary prohibitions without making it so cautious that it refuses to answer harmless questions, or so confident that it accidentally breaks the rules. If a model fails to follow these instructions, it can lead to leaked data or brand safety incidents; if it becomes too afraid to speak, it becomes useless.

Researchers at Tencent have developed a new method called DUET to solve this specific problem of teaching models to follow runtime rules. Instead of trying to retrain the model from scratch or simply showing it examples of good and bad answers, they created a system that acts like a pair of identical twins. One twin sees the forbidden rule in its instructions, while the other twin, who has the exact same knowledge and abilities, does not see the rule. Because these two "teachers" are otherwise identical, any difference in what they say must be caused solely by the presence or absence of that specific rule. The researchers use this disagreement to pinpoint exactly where the model is likely to make a mistake.

The process works by having the student model generate a response while both teacher twins watch. When the two teachers agree on what the next word should be, the system ignores that part of the conversation, assuming it is safe and unimportant. However, when the teachers disagree—one suggesting a word that reveals a secret and the other suggesting a safe alternative—the system flags that specific moment as a critical learning opportunity. This allows the student model to focus its attention only on the few words where the rule actually matters, rather than wasting effort on the thousands of words that are perfectly fine. The student is then gently pushed away from the teacher who ignores the rule and pulled toward the teacher who respects it, effectively learning to navigate the boundary between helpfulness and compliance.

This approach addresses a flaw in older methods where models were trained on entire conversations. In those older systems, if a model made a single mistake in a long answer, the whole answer was often treated as bad, which confused the model about which specific words were the problem. By isolating the disagreement to the exact token, or word unit, where the violation occurs, the new method provides a much cleaner signal. The researchers tested this on a variety of tasks, including protecting personal information, following safety guidelines, and adhering to business tool definitions. They found that the models trained with this dual-teacher method were significantly better at refusing to break rules while still remaining helpful for legitimate questions.

The results showed that the new method outperformed existing techniques across different sizes of models, from smaller ones with 1.5 billion parameters to larger ones with 8 billion. In their tests, the models achieved a compliance rate of over 72% to 85% on violating requests, meaning they successfully refused to break the rules, while still maintaining a high level of usefulness for normal questions, preserving between 88% and 93% of their ability to answer correctly. Crucially, the models did not lose their general intelligence; they remained just as good at math and logic puzzles as before. The study suggests that by using two identical models to find the exact spot where a rule matters, developers can create AI assistants that are both safe and useful, navigating complex, changing instructions without needing massive amounts of new data or losing their core capabilities.

Technical Summary: DUET – Dual-Teacher On-Policy Distillation for Prohibition Compliance

1. Problem Definition: Prohibition Compliance

The paper addresses a specific challenge in real-world Large Language Model (LLM) deployments: Prohibition Compliance. Unlike general safety alignment, this task involves runtime-injected prohibitions (e.g., enterprise policies, PII redlines, tool boundaries) embedded in the system prompt that vary per request and tenant. The core difficulty lies in three structural features:

Runtime Configurability: The prohibition resides in the prompt, not the model weights.
Localized Violation: Only a small subset of tokens in a long response actually violates the rule, while the rest of the response may be compliant.
Joint Compliance-Utility Requirement: The model must refuse violating requests without over-refusing legitimate queries within the same business context.

The authors argue that conventional post-training paradigms are structurally ill-suited for this:

Supervised Fine-Tuning (SFT): Hides the violation signal within compliant labels, failing to explicitly teach the model where and how to suppress specific violating tokens.
Direct Preference Optimization (DPO): Uses sequence-level chosen/rejected labels, which condemns an entire sequence for a few offending tokens. This contaminates neutral positions and dilutes the correction signal at the true violation points.
Vanilla On-Policy Distillation (OPD): While it corrects along the student's actual failure trajectories, it suffers from two gaps: (1) the teacher's logits are conditioned on the student's potentially erroneous prefix, causing the teacher to "drift" toward the violation and provide misleading supervision; and (2) standard token-mean losses waste supervision on neutral tokens and dilute it on critical violation tokens.

2. Methodology: DUET Framework

The proposed solution, DUET (DUal-tEacher Token-selective OPD), introduces a dual-teacher architecture to isolate the causal effect of the prohibition and drive token-granular optimization.

Core Mechanism: Same-Weight Disagreement
DUET employs two teachers, $T^+$ and $T^-$ , which share identical weights but differ only in their input prompts:

$T^+$ (Positive Teacher): Sees the prohibition in the system prompt (and includes a "thinking prefix" to encourage compliance).
$T^-$ (Negative Teacher): Does not see the prohibition and lacks the thinking prefix.

Because the weights are identical, any per-token disagreement between $T^+$ and $T^-$ is solely attributable to the presence of the prohibition, effectively filtering out intrinsic model capacity gaps or distributional mismatches.

Two-Stage Process:

Violation-Token Selection (Signal Cleaning):
- The method discards tokens where $T^+$ and $T^-$ agree. Agreement implies either redundant supervision (ordinary tokens) or misleading supervision (where the student's prefix has dragged $T^+$ toward $T^-$ ).
- Only tokens with high KL disagreement are retained.
- A three-stage filter selects the top- $K$ disagreeing tokens per sequence and applies max-pool dilation to capture short sub-word clusters of violations.
Preference-Directed Learning:
- On the selected high-disagreement tokens, the student is optimized using a token-granular preference objective that embeds DPO-style logic directly into OPD without offline preference data.
- Pull ( $L_{pos}$ ): Pulls the student distribution toward $T^+$ .
- Push ( $L_{neg}$ ): Pushes the student away from $T^-$ by suppressing candidates that $T^-$ prefers over $T^+$ .
- Synthesized Target ( $L_{syn}$ ): To further purify the signal, a synthesized target $z_{adj}$ is created by extrapolating $T^+$ further away from $T^-$ in logit space, stripping residual contamination.
- Utility Preservation ( $L_{util}$ ): To prevent over-refusal on legitimate queries, a top- $K$ knowledge distillation term is applied against $T^+$ on non-violating (normal) queries, ensuring the model remains helpful.

The total loss combines these terms: $L = \lambda_{pos} L_{pos} + \lambda_{neg} L_{neg} + \lambda_{syn} L_{syn} + \lambda_{util} L_{util}$ .

3. Key Contributions

DUET Framework: A novel dual-teacher token-selective OPD method that uses same-weight teacher disagreement to isolate prohibition signals, enabling online signal cleaning and token-granular preference optimization without offline preference data.
Industrial Benchmark: The construction of a Prohibition-Compliance benchmark spanning five task families (PII/memory, safety redlines, custom business content, tool definitions, business SOPs). This benchmark uniquely measures explicit-violation refusal, paraphrase robustness, and over-refusal risk on legitimate queries within the same business context.
Systematic Evaluation: Comprehensive validation across Qwen2.5 and Qwen3 model families (1.5B to 8B scales), demonstrating significant improvements over strong distillation baselines.

4. Experimental Results

The authors evaluated DUET on their industrial benchmark and public constraint-following benchmarks (SysBench, GSM8K, MATH-500).

Prohibition Compliance: Across 1.5B–8B Qwen variants, DUET achieved 72.3–85.2% violation compliance while preserving 88–93% normal utility.
- It significantly outperformed the teacher model and other baselines (OPD-F, OPD-R, AlignDistil, TIP).
- On the Qwen2.5-7B $\to$ 1.5B distillation, DUET improved violation refusal from ~51% (best baseline) to 72.3%, while maintaining 93% utility.
- The paper notes that gains are larger when the teacher is "dirtier" (lower inherent compliance) or when there is a larger vocabulary mismatch between teacher and student, as DUET's disagreement signal effectively filters out these confounders.
General Capability: DUET training resulted in no catastrophic forgetting on general reasoning benchmarks. On GSM8K and MATH-500, the model maintained performance comparable to the baseline, with only minor fluctuations in specific sub-domains (Number Theory, Counting) attributed to pre-existing student weaknesses.
Constraint Following (SysBench): DUET significantly outperformed all distillation baselines on the SysBench (CSR and ISR metrics), despite being trained on data that did not contain SysBench-style prompts. This suggests the learned alignment is a general prohibition-compliance capability rather than template memorization.
Ablation Studies: Removing any component (e.g., the synthesized target $L_{syn}$ or the utility term $L_{util}$ ) led to significant drops in either violation refusal or utility, confirming the necessity of each design element.

5. Significance and Claims

The paper claims that DUET addresses a critical gap in industrial LLM deployment where runtime-configurable rules require precise, token-level control without sacrificing utility. By leveraging the same-weight disagreement between a positive and negative teacher, DUET provides a clean, online supervision signal that is uncontaminated by model capacity issues or prefix pollution.

The authors position DUET as a solution that moves beyond the limitations of sequence-level preference optimization and vanilla on-policy distillation. They emphasize that their method allows for token-granular DPO-style optimization directly within the on-policy distillation loop, eliminating the need for costly offline preference data curation while effectively handling the "joint compliance-utility" requirement inherent in real-world business scenarios.