🤖 machine learning

On-Policy Delta Distillation

This paper introduces On-Policy Delta Distillation (OPD $^2$ ), a novel post-training method for reinforcement learning that leverages a "delta signal"—representing the difference between a teacher model and its base prior to reasoning tuning—to more effectively transfer reasoning capabilities and achieve strong performance across mathematics, science, and code benchmarks with minimal post-training.

Original authors: Byeongho Heo, Jaehui Hwang, Sangdoo Yun, Dongyoon Han

Published 2026-07-17

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Byeongho Heo, Jaehui Hwang, Sangdoo Yun, Dongyoon Han

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers are like giant, hungry libraries that have read almost every book ever written. These libraries, called Large Language Models (LLMs), are incredibly smart at guessing the next word in a sentence. But just knowing how to finish a sentence isn't enough to solve a tricky math problem or debug a broken computer program. To get really good at those tasks, we need to teach them how to think.

Think of this process like training a student. First, the student reads a massive library (pre-training) to learn the basics of language. Then, a teacher comes in to give them specific lessons on how to solve puzzles (post-training). Usually, teachers use two main methods: they either show the student the perfect answer and say, "Copy this" (Supervised Fine-Tuning), or they let the student try, give them a grade, and say, "Try again, but do better next time" (Reinforcement Learning). Recently, a new method called "On-Policy Distillation" became popular. It's like a teacher watching the student solve a problem in real-time and whispering, "Yes, that word was good," or "No, that word was wrong," based on what the teacher would have said. It's efficient and avoids the messy business of designing complex grading systems.

But here's the catch: even this smart method has a flaw. When the teacher whispers, they might accidentally whisper things the student already knows, like how to be polite or how to tell a story, rather than the specific "aha!" moments of logic that make the teacher a genius at reasoning. The paper you're about to read asks a simple question: What if we could filter out the noise and only teach the student the new tricks the teacher learned?

This paper introduces a clever new technique called On-Policy Delta Distillation (OPD2). The researchers realized that a "Reasoning Teacher" is actually two models in one: the original "Base" model (before it learned to reason) and the "Reasoning" model (after it learned). If you subtract the Base model's thoughts from the Reasoning model's thoughts, you get a "Delta Signal." This signal is like a highlight reel of exactly what changed when the teacher learned to think. Instead of just copying the teacher's final answer, the student learns from this "Delta Signal"—the specific difference that represents the leap in reasoning ability.

The authors tested this idea by mixing math, science, and coding problems. They found that using this "Delta Signal" as the main reward helped students learn much faster and better than the old methods. In fact, their new method, OPD2, consistently beat the previous best techniques across different sizes of models, from tiny 1.7-billion-parameter models to massive 8-billion ones. It worked so well that a smaller model trained with OPD2 could sometimes outperform a much larger model trained with older methods. The paper suggests that by focusing only on the change in thinking, rather than the whole personality of the teacher, we can teach AI to reason more effectively without wasting time on what it already knows.

The Story of the "Delta" Signal

Let's dive into the mechanics of this discovery using a simple analogy. Imagine you are learning to play a complex video game. You have a "Base" version of your character that knows how to walk, jump, and talk. Then, you get a "Pro" version of that character who has mastered the game's hardest levels.

The old way of teaching (standard On-Policy Distillation) is like the Pro character standing next to you and saying, "Do exactly what I do." The problem is, the Pro character still walks and talks exactly like the Base character did. So, when the Pro says, "Walk forward," they are just repeating something you already know how to do. You spend your time practicing walking, not learning the secret combos that make you a pro.

The authors of this paper, Byeongho Heo, Jaehui Hwang, Sangdoo Yun, and Dongyoon Han from NAVER AI Lab, proposed a different approach. They asked: "What if we only teach the student the difference between the Pro and the Base?"

This difference is the Delta Signal.

The Base Model: Knows how to speak and write, but might stumble on complex logic.
The Reasoning Teacher: Knows how to speak, write, and solve hard logic puzzles.
The Delta: The specific "magic" the Teacher learned to solve the puzzle.

In the paper, they visualize this with word clouds. When they looked at what the old method taught, it was full of generic words like "see," "try," and "verify." But when they looked at the Delta Signal, the words that stood out were logical connectors like "hence," "however," "note," and "instead." These are the words that glue a logical argument together. The Delta Signal effectively filters out the "boring" stuff the student already knows and highlights the "secret sauce" of reasoning.

How They Built OPD2

To make this work, the researchers had to solve a few tricky problems. If you just give a student the Delta Signal, they might get confused because the signal doesn't account for what the student themselves is doing. It's like a coach shouting instructions without watching the player's current position.

So, they added two special ingredients to their recipe:

Centering: They adjusted the signal so it's balanced around zero. This helps the student understand whether a move is "better than average" or "worse than average," rather than just getting a raw score that might be misleading.
Joint Conditioning: They made sure the new Delta Signal only kicks in when it agrees with the old, standard method. This acts like a safety net. If the Delta Signal tries to push the student in a weird direction that contradicts the teacher's general style, the system says, "Hold on," and stops the update. This prevents the student from getting confused or forgetting how to speak properly while trying to learn how to think.

The Results: A New Champion

The team tested their new method, which they named OPD2, against the old standard (OPD) and a newer, advanced method called ExOPD. They used a mix of 100,000 questions from math, science, and coding datasets. They tested this on various models, including the Qwen3 family (sizes 1.7B, 4B, and 8B) and the Gemma4 model.

The results were impressive. In the "non-thinking" mode (where models answer quickly without showing their work), OPD2 consistently outperformed the others.

For the Qwen3-1.7B model in math, OPD2 boosted the average score from 34.8 to 54.6. That's a huge jump, beating the next best method by over 3 points.
For the Qwen3-4B model, OPD2 reached an average score of 70.3, while the previous best (ExOPD) was at 66.4.
Even more surprisingly, the 4B model trained with OPD2 performed better than the 8B model trained with the old methods. This suggests that how you train matters as much as how big your model is.

They also tested the models in "thinking" mode, where the models are already quite smart and show their reasoning steps. This is harder to improve because the models are already good. Yet, OPD2 still managed to squeeze out extra performance. For example, on the HMMT25 math benchmark, the Qwen3-8B model improved its score from 44.3 to 52.3 with OPD2, while the other methods actually made the score worse or barely changed it.

The method also worked on Gemma4, a different family of models. While other methods struggled or even made the model worse, OPD2 improved the math scores from 60.6 to 67.8. This shows that the idea of using the "Delta Signal" isn't just a fluke for one specific model; it seems to be a general principle that helps AI learn to think better.

Why This Matters

The paper suggests that the key to teaching AI to reason isn't just feeding it more data or making the teacher bigger. It's about being precise about what we are teaching. By isolating the specific changes that happen when a model learns to reason (the Delta), we can transfer that ability much more efficiently.

The authors note that this method is computationally efficient. It takes about 24-28% more time to train than the standard method because the computer has to run the "Base" model in the background to calculate the difference. However, since these training sessions are usually short (often less than one full pass through the data), this small extra cost is worth the massive performance gain.

In short, OPD2 is like a master chef who realizes that to teach an apprentice the secret to a perfect soup, they shouldn't just say "add salt." They should say, "Add this much more salt than you usually do, because that's the difference between a good soup and a great one." By focusing on the difference, the student learns the secret faster and better. The paper concludes that this approach is a significant step forward in making AI smarter, faster, and more capable of solving the complex problems of the future.

Technical Summary: On-Policy Delta Distillation (OPD²)

1. Problem Statement

On-Policy Distillation (OPD) has emerged as a promising post-training alternative to Reinforcement Learning (RL) for aligning Large Language Models (LLMs) with reasoning capabilities. Unlike traditional Knowledge Distillation (KD) which often relies on teacher-generated sequences (SFT), OPD trains a student model on its own generated sequences, using the teacher model to score sampled tokens. This approach mitigates exposure bias and reduces the computational and design complexities associated with reward modeling in RL.

However, the fundamental design of the OPD reward signal remains under-explored. Standard OPD defines the reward as the log-probability difference between the teacher ( $\pi^*$ ) and the student ( $\pi_\theta$ ). The authors argue that this approach forces the student to mimic the teacher's entire output distribution, including natural linguistic preferences and stylistic tendencies acquired before the teacher underwent reasoning-specific tuning. Consequently, the student may fail to isolate and effectively transfer the specific "learning trajectory" or reasoning capabilities acquired during the teacher's post-training phase.

2. Methodology: On-Policy Delta Distillation (OPD²)

The paper proposes On-Policy Delta Distillation (OPD²), a novel framework that redefines the distillation reward signal to focus exclusively on the reasoning knowledge gained by the teacher.

2.1 The Delta Signal

Instead of comparing the student to the teacher, OPD² introduces a delta signal ( $\Delta$ ), defined as the difference between the reasoning-tuned teacher model ( $\pi^*$ ) and its base model prior to instruction tuning ( $\pi^*_{\text{base}}$ ).

The reward for a token $y_t$ is formulated as:
$R^\Delta_t = \log \pi^*(y_t | x, y_{<t}) - \log \pi^*_{\text{base}}(y_t | x, y_{<t})$

This signal captures the specific changes induced by reasoning tuning, effectively filtering out the base model's inherent preferences. The authors hypothesize that this isolates the "reasoning knowledge" required for complex tasks, rather than general language modeling capabilities.

2.2 Reward Design and Stabilization

Directly applying the delta signal presents challenges, specifically regarding convergence. Since $R^\Delta_t$ does not depend on the student's probability $\pi_\theta$ , training based solely on this signal could lead to instability or convergence to a one-hot vector maximum. To address this, the authors introduce two design mechanisms:

Centering: Similar to Advantage functions in RL, the expected reward under the student's policy is subtracted to handle signal bias:
$A^\Delta_t = R^\Delta_t - \mathbb{E}_{\tilde{y}_t \sim \pi_\theta} [R^\Delta_t]$
Joint Conditioning: To prevent the student from diverging too far from the teacher's actual distribution (which could degrade performance), the authors introduce a conditional stop mechanism. The final advantage function $A^{D2}_t$ is active only when the delta signal aligns with the original OPD signal direction:
$A^{D2}_t = \begin{cases} A^\Delta_t & \text{if } A^\Delta_t \cdot A^{OPD}_t > 0 \\ 0 & \text{otherwise} \end{cases}$
This ensures that updates occur only when the delta signal reinforces the direction of the standard distillation signal, preserving the student's original capabilities while enhancing reasoning.

3. Key Contributions

Novel Reward Formulation: The introduction of the delta signal ( $\pi^* - \pi^*_{\text{base}}$ ) as the primary learning objective for on-policy distillation, shifting focus from mimicking the teacher to mimicking the teacher's improvement.
Theoretical and Empirical Analysis: The paper provides word cloud visualizations, token-level signal analysis, and statistical evidence demonstrating that the delta signal emphasizes logical connectives (e.g., "hence," "however") and suppresses generic exploratory phrases (e.g., "see," "try") more effectively than standard OPD.
Stabilization Mechanisms: The proposal of centering and joint conditioning to ensure stable convergence when using the delta signal.
Extensive Benchmarking: A comprehensive evaluation framework across three reasoning domains (Math, Science, Code) and multiple model families (Qwen3, Gemma4) with varying sizes (1.7B to 30B).

4. Experimental Results

The authors evaluated OPD² against standard OPD and ExOPD (a recent advanced method) on 14 benchmarks.

Non-Thinking Mode: OPD² consistently outperformed baselines. For Qwen3-1.7B, the average Math score improved from 34.8 (base) to 54.6 (OPD²), surpassing ExOPD (51.4). Notably, a 4B model trained with OPD² outperformed an 8B model trained with other methods, suggesting the method can compensate for model size.
Thinking Mode: In scenarios where baseline models already possessed strong reasoning capabilities, standard OPD often degraded performance. OPD², however, consistently improved or maintained performance across all model sizes and domains, achieving the highest average scores in Math, Code, and Science.
Cross-Architecture Generalization: When applied to Gemma4-E4B-it, standard OPD caused significant performance drops. OPD² successfully generalized, improving Math performance by a clear margin (60.6 to 67.8) and retaining coding capabilities better than other methods.
Training Dynamics: OPD² demonstrated a persistent performance advantage throughout the training trajectory, whereas OPD and ExOPD often plateaued or degraded after initial gains.
Ablation Studies: Removing the delta signal caused the most significant performance drop, confirming it as the primary driver of improvement. The centering and joint condition components provided additional, though smaller, stability and performance gains.
Computational Cost: OPD² incurs a modest overhead (approx. 24-28% increase in wall-clock time for Qwen3 models) due to the additional forward pass of the base model. This remains comparable to ExOPD and is considered acceptable given the short training regimes typical of on-policy distillation.

5. Significance and Claims

The paper claims that OPD² represents a fundamental improvement in the design of on-policy distillation. By utilizing the delta signal, the method successfully isolates and transfers the specific reasoning capabilities acquired during a teacher's post-training phase, rather than just its general output distribution.

The authors position OPD² as a robust alternative to RL that:

Achieves strong reasoning performance with short post-training periods.
Generalizes effectively across different model sizes and architectures (Qwen and Gemma).
Avoids the performance degradation often seen when applying standard distillation to already strong "thinking" models.

The work suggests that focusing on the change in model behavior (the delta) rather than the absolute behavior is a more effective strategy for transferring complex reasoning skills, marking a potential milestone in the development of efficient post-training methods for LLMs.