🤖 AI

DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models

The paper introduces DASH (Divergence-Adaptive Supervision Horizons), a method that enhances on-policy self-distillation for reasoning models by adaptively adjusting token-level supervision weights based on the temporal evolution of teacher-student divergences, thereby improving performance across mathematical benchmarks without requiring additional forward passes.

Original authors: ZhiYan Hou, Xinyu Tang, Hongyan An, Jianjin Zhang, Weizhen Wang, Yunyun Han, Gengsheng Li, Xiangzhao Hao, Haiyun Guo, Wenbin Hu, Jinqiao Wang, Yafeng Deng

Published 2026-08-07

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: ZhiYan Hou, Xinyu Tang, Hongyan An, Jianjin Zhang, Weizhen Wang, Yunyun Han, Gengsheng Li, Xiangzhao Hao, Haiyun Guo, Wenbin Hu, Jinqiao Wang, Yafeng Deng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers are learning to solve complex puzzles, like advanced math problems or writing computer code, by practicing over and over again. This field is called Reinforcement Learning, where the computer (the "student") tries different answers and gets a simple "yes" or "no" at the very end to see if it was right. The problem is, if the student takes a long, winding path to get there, that single "yes" or "no" doesn't tell them which specific steps were good and which were bad. It's like getting a "B" on a final exam without knowing which questions you missed. To fix this, researchers use a trick called "On-Policy Self-Distillation." Think of this as the student asking a "privileged teacher" (who knows the secret answer key) to grade every single step of the student's journey as they go. This gives the student a dense stream of feedback instead of just one at the end. But here's the catch: the standard way of using this teacher treats every single step of feedback exactly the same, as if a mistake made at the very beginning is just as important as a mistake made at the very end, regardless of how the student's performance was changing along the way.

This paper introduces a new method called DASH (Divergence-Adaptive Supervision Horizons) to fix that blind spot. The researchers found that the standard approach is too rigid; it doesn't pay attention to the story of the student's mistakes. Sometimes, a small error early on might lead to a total disaster later, while a similar error later might be harmless. DASH acts like a smart editor that looks at the whole sequence of feedback. If the student is drifting away from the teacher's advice in a way that suggests a growing problem, DASH amplifies the weight of those signals to correct the course. If the student is doing fine, it tones them down. The paper shows that by making these adjustments, DASH helps the student learn much faster and better than before, improving their math reasoning scores across different model sizes without needing any extra computing power or a smarter teacher.

The Problem: The "One-Size-Fits-All" Grader

In the world of training AI to reason, we have a student model and a teacher model. The student generates a long chain of thoughts (a "rollout") to solve a problem. The teacher, who has access to the correct solution, watches the student and says, "Hey, at this specific word, you should have chosen this other word." This is called token-level supervision.

The standard method, called OPSD (On-Policy Self-Distillation), collects all these little "you're wrong" signals from the teacher and averages them out. It's like a teacher giving a student a report card where every single mistake counts exactly the same, no matter when it happened.

The authors realized this is a bit silly. Imagine a student solving a math problem.

Scenario A: The student makes a tiny calculation error in step 1, but then catches themselves and fixes it in step 2. The "divergence" (the gap between student and teacher) was high for a moment, but then it went away.
Scenario B: The student makes the exact same tiny error in step 1, but then keeps making it worse and worse, spiraling into a completely wrong answer by step 10.

In standard OPSD, both scenarios get the exact same "penalty" for that first step. The system doesn't realize that in Scenario B, that first error was the start of a train wreck, while in Scenario A, it was just a minor stumble. The standard method treats the local mistake in isolation, ignoring the history of how the student performed leading up to it and the future of where that mistake might lead.

The Solution: DASH, the Adaptive Coach

The authors propose DASH to give the grading system a memory and a sense of context. Instead of just averaging the mistakes, DASH asks: "How does this specific mistake fit into the story of the whole answer?"

Here is how DASH works, using a simple analogy:

Imagine the student is walking a path, and the teacher is holding a rope attached to the student's waist. The "divergence" is how far the student is pulling away from the teacher's ideal path.

Measuring the Gap: DASH looks at how far the student is from the teacher at every single step.
The "Gate" Mechanism: Instead of just pulling the student back with the same force every time, DASH uses a special "gate."
- If the student is doing worse than average (the gap is getting bigger), the gate opens wide. This means the teacher's feedback for future steps gets a stronger signal to pull the student back hard. It says, "Hey, this is a bad trend! We need to pay extra attention to fixing this path."
- If the student is doing better than average (the gap is small), the gate closes a bit. The teacher's feedback is still there, but it doesn't scream as loudly because the student is already on the right track.
Looking Backward: DASH uses a clever trick called "backward aggregation." It starts at the end of the answer and works its way back to the beginning. It asks, "If I look at the end of this path, how much did this specific step contribute to the final mess (or success)?" It then adjusts the weight of the feedback for that step based on what happened after it.

This creates a dynamic system where the "importance" of a mistake isn't fixed. A mistake that leads to a long string of errors gets a heavy weight. A mistake that gets corrected quickly gets a lighter weight.

What They Found

The researchers tested this idea on three different sizes of AI models (1.7 billion, 4 billion, and 8 billion parameters) using three tough math benchmarks (AIME 2024, AIME 2025, and HMMT February 2025).

The Results: DASH consistently beat the standard method (OPSD) across the board. For the smallest model (1.7B), the average accuracy jumped from 41.87% to 45.07%. For the larger models, the gains were also clear, with the 8B model going from 64.80% to 66.40%.
The "Why": Through a series of experiments, they proved that the improvement wasn't just because they added more "noise" or changed the math randomly. They tested what would happen if they used a fixed gate (like the old method) versus a smart, changing gate. The smart gate (DASH) was the winner. They also checked if the direction of the logic mattered (e.g., should we punish big gaps more or less?). They found that their specific logic—punishing big gaps by opening the gate wider to let in more correction—was the key.
Efficiency: The best part is that DASH doesn't need a super-smart teacher or extra computing power. It just reuses the information the standard method was already calculating, but processes it in a smarter way. It's like taking a pile of raw data and organizing it better, rather than gathering more data.

The Takeaway

The paper suggests that when teaching AI to reason, we shouldn't just count mistakes; we should understand the story of the mistakes. By making the supervision "adaptive"—changing how much we listen to the teacher based on how the student is doing over time—we can help the AI learn much more effectively. DASH shows that a little bit of context goes a long way, turning a rigid grading system into a flexible, intelligent coach that knows when to be strict and when to be lenient.

Technical Summary: DASH - Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has improved the reasoning capabilities of Large Language Models (LLMs) by utilizing automatically verifiable signals (e.g., answer correctness). However, standard RLVR relies on sparse, sequence-level rewards, creating a "temporal credit assignment" challenge where it is difficult to distinguish the contribution of individual token-level decisions to the final outcome.

To mitigate this, On-Policy Self-Distillation (OPSD) has been introduced. In OPSD, a student model generates a response, while the same model, conditioned on privileged information (such as a reference solution), acts as a teacher to provide dense, token-level distributional supervision.

The Core Limitation: While OPSD alleviates signal sparsity, the paper identifies that standard ("vanilla") OPSD underutilizes the temporal structure of the rollout. Vanilla OPSD aggregates local distributional discrepancies (divergences) using uniform coefficients ( $1/T$ ) for every token, regardless of:

The token's position.
The history of discrepancies leading up to that token.

The authors argue that in autoregressive generation, the same local divergence magnitude can occur in vastly different temporal contexts (e.g., after a long history of correct reasoning vs. after a series of errors). Since the local scalar alone cannot distinguish these contexts, assigning a uniform weight prevents the supervision signal from adapting to the realized discrepancy sequence.

2. Methodology: Divergence-Adaptive Supervision Horizons (DASH)

To address the inability of vanilla OPSD to adapt to temporal discrepancy profiles, the authors propose DASH. The method introduces a mechanism to dynamically adjust token-level supervision weights based on how local divergences evolve during generation.

Key Components

Sequence-Relative Divergence Gaps:
Instead of using raw local divergence values ( $d_t$ ) directly, DASH computes the gap between the local distillation signal and the sequence-level mean ( $\bar{r}$ ):
$g_t = r_t - \bar{r}$
where $r_t$ is the clipped local KL divergence at step $t$ .
Adaptive Propagation Gates ( $\lambda_t$ ):
These gaps are mapped to adaptive propagation gates using a sigmoid function with a sensitivity parameter $\kappa$ :
$\lambda_t = \text{sg}[\sigma(-\kappa g_t)]$
- Logic: If the local divergence is below the sequence mean ( $g_t < 0$ ), the gate $\lambda_t$ is larger, allowing future signals to propagate further back. If the divergence is above the mean ( $g_t > 0$ ), the gate is smaller, restricting propagation. This creates a "supervision horizon" that adapts to the sequence.
Backward Multi-Step Aggregation:
DASH replaces the uniform aggregation of vanilla OPSD with a backward recursion that constructs a path-dependent aggregated signal ( $A_t$ ):
$A_T = r_T$
$A_t = r_t + \lambda_t A_{t+1} \quad \text{for } t = T-1, \dots, 1$
The final loss is the average of these aggregated signals:
$L_{\text{DASH}} = \frac{1}{T} \sum_{t=1}^T A_t$

Structural Insight

The authors derive a fixed-horizon gradient decomposition for the expected objective, showing that the exact gradient contains a trajectory term where coefficients depend on future divergences. While DASH does not explicitly estimate future divergences or use score-function gradients, it mimics this order-dependent structure by using the realized discrepancy sequence to condition the weights of the direct distillation term.

3. Key Contributions

The paper outlines three primary contributions:

Identification of a Temporal Gap: The authors characterize a limitation in vanilla OPSD where assigning uniform weights to local divergences makes the objective invariant to different temporal arrangements of the same divergence values, preventing adaptation to temporal structure.
Proposal of DASH: A new method that uses sequence-relative divergence gaps to construct adaptive propagation gates. This enables dense distillation supervision to exploit the temporal evolution of local discrepancies through multi-step aggregation.
Empirical Validation: Comprehensive experiments across three mathematical reasoning benchmarks (AIME 2024, AIME 2025, HMMT 2025) and three model scales (Qwen3-1.7B, 4B, 8B) demonstrate that DASH consistently outperforms matched vanilla OPSD reruns and other recent distillation baselines.

4. Experimental Results

The authors conducted experiments using Qwen3 Instruct models (1.7B, 4B, 8B) on mathematical reasoning tasks.

Performance Gains: DASH achieved the highest scores in all nine benchmark–model settings compared to baselines (Base, SFT, GRPO, OPSD, EOPD, AVSD, PW-OPSD).
- On Qwen3-1.7B, DASH improved the average accuracy from 41.87% (OPSD) to 45.07% (+3.20 points).
- On Qwen3-4B, it improved from 63.60% to 65.00% (+1.40 points).
- On Qwen3-8B, it improved from 64.80% to 66.40% (+1.60 points).
Ablation Studies:
- Adaptive vs. Fixed: Replacing adaptive gates with fixed coefficients ( $\lambda$ ) improved performance over vanilla OPSD but remained inferior to DASH, confirming that adapting to the specific discrepancy sequence is crucial.
- Gate Direction: Reversing the sign of the gate function (Inverse-gap) resulted in poor performance, validating the specific design hypothesis that propagation should remain open after small discrepancies and restrict after large ones.
- Coefficient Allocation vs. Scale: Disentangling the relative coefficient profile from the average scale showed that the primary gains come from the discrepancy-conditioned allocation, not merely an increase in the overall supervision scale.
Design Choices:
- Sensitivity ( $\kappa$ ): A moderate sensitivity ( $\kappa=5$ ) yielded the best results, balancing insufficient adaptation and overly sharp responses.
- Divergence Metric: Forward KL divergence outperformed symmetric JSD and Reverse KL.
- Vocabulary Support: Full vocabulary support was optimal; compressing to top-1 tokens caused significant performance drops.

5. Significance and Claims

The paper claims that DASH offers a significant improvement in on-policy self-distillation by leveraging the temporal structure of teacher-student discrepancies without requiring additional computational overhead.

Efficiency: DASH reuses the teacher and student distributions already computed by standard OPSD. It requires no additional teacher or student forward passes, making the gains computationally efficient.
Generalizability: The improvements were consistent across different model scales and benchmarks, suggesting the method is not confined to specific model capacities or datasets.
Insight: The work provides new insights into how dense supervision should be allocated in long-horizon reasoning, moving beyond uniform weighting to sequence-aware, adaptive horizons.

The authors conclude that by adapting token-level weights to the realized discrepancy sequence, DASH effectively addresses the temporal credit assignment challenge inherent in standard RLVR and OPSD approaches.