Technical Summary: Finding the Evidence for On-Policy Reasoning Distillation

1. Problem Statement

On-policy distillation (OPD) has emerged as a dominant paradigm for teaching small models to reason by providing dense token-level supervision from a teacher model. However, current OPD methods face a critical limitation: they fail to distinguish between different types of knowledge within reasoning chains. Training on all tokens dilutes the learning signal, while recent entropy-based selection methods (e.g., Wang et al., 2026) only capture "decision tokens" (where the student is uncertain) but miss "evidence tokens."

The paper identifies a structural blind spot in existing methods:

• Decision Tokens: Positions where the student faces genuine uncertainty (e.g., branching points, strategy choices). These are naturally identified by high student entropy.
• Evidence Tokens: Intermediate steps that justify decisions (e.g., derivations, variable substitutions). These often appear where the student is confident yet wrong. Because the student is confident, entropy is low, rendering these tokens invisible to entropy-based selectors. Consequently, substantive reasoning knowledge remains untransferred, leading to students that know when to branch but fail at how to execute the branch correctly.

2. Methodology: DEAR (Decision-Evidence Aware Reasoning Distillation)

The authors propose DEAR, a two-stage token selection mechanism designed to recover both decision and evidence tokens for OPD training.

Stage 1: Decision Identification

The method first identifies decision tokens using student entropy ($H^S_t$). The top-$p\%$ tokens with the highest entropy are selected as the decision set ($D$). These tokens mark the "skeleton" of the reasoning process where the student must make a choice.

Stage 2: Evidence Discovery

To find the supporting evidence tokens that entropy misses, DEAR leverages the relationship between decisions and their surrounding context:

1. Contextual Similarity: The method extracts the student's hidden states ($h^L_t$) from the deepest transformer layer. For every non-decision token, it computes the cosine similarity to the nearest decision anchor (any token in set $D$). High similarity indicates the token participates in the same reasoning context (i.e., it is part of the derivation supporting a decision).
2. Knowledge Gap Boosting: Not all contextually similar tokens require correction. To prioritize tokens where the student lacks knowledge, the method incorporates teacher–student divergence ($\delta_t = |\log \pi_\theta(y_t) - \log \pi_T(y_t)|$).
3. Scoring: An Evidence Salience Score ($s_j$) is calculated for non-decision tokens:
   $$s_j = \hat{a}_j \times (1 + \hat{\delta}_j)$$
   Where $\hat{a}_j$ is the normalized cosine similarity (acting as a gate to filter out noise) and $\hat{\delta}_j$ is the normalized divergence (boosting tokens with large knowledge gaps).
4. Selection: The top-$q\%$ of non-decision tokens with the highest scores form the evidence set ($E$).

Final Training Objective

The final training set $S$ is the union of decisions and evidence ($S = D \cup E$). The OPD loss is computed exclusively over these selected tokens, ensuring the model learns both the branching strategy and the substantive intermediate steps.

3. Key Contributions

1. Decomposition of Reasoning Chains: The paper empirically demonstrates that reasoning chains consist of two distinct knowledge types requiring different discovery mechanisms: decisions (high entropy) and evidence (low entropy, high divergence).
2. Discovery Mechanism: It introduces a method to discover "confidently wrong" evidence tokens by combining hidden-state cosine similarity (to link tokens to decisions) with divergence boosting (to prioritize knowledge gaps). This overcomes the structural impossibility of entropy-based selection finding these tokens.
3. Performance Gains: DEAR consistently outperforms standard OPD and offline knowledge distillation across multiple benchmarks.

4. Experimental Results

The authors evaluated DEAR across three student-teacher configurations (varying model families and sizes) on six mathematical reasoning benchmarks and three code generation benchmarks.

• Mathematical Reasoning:
  • DEAR achieved up to +2.5 percentage points (pp) improvement on competition-level math (e.g., AIME) compared to standard OPD.
  • Gains were most pronounced in "cross-family" distillation (Setting B: Qwen2.5-1.5B student, Qwen3-4B teacher), where the knowledge gap is largest.
  • On AIME 2024 (Setting A), DEAR reached 4.58% accuracy vs. 2.08% for standard OPD (a 2.2$\times$ improvement).
• Code Generation:
  • DEAR improved pass rates by +5.74pp on MBPP, +2.13pp on HumanEval, and +2.56pp on APPS.
  • Notably, standard OPD degraded performance on MBPP compared to the base model, whereas DEAR resolved this, suggesting it effectively filters noise in code generation.
• Signal Analysis:
  • Entropy-only selection captured only ~39% of the total gradient mass.
  • DEAR's inclusion of evidence tokens increased gradient mass capture to ~76%, nearly doubling the effective learning signal.
  • Analysis confirmed that evidence tokens are concentrated in the "low-entropy, high-divergence" region, which entropy-based methods systematically ignore.

5. Significance and Claims

The paper claims that complete reasoning distillation requires transferring not just where to decide, but the knowledge that justifies those decisions.

• Beyond Entropy: The work challenges the prevailing assumption that high-entropy tokens are the sole carriers of reasoning ability. It demonstrates that "confident errors" contain critical, untransferred knowledge.
• Structural Insight: By identifying that evidence tokens cluster around decisions in the hidden state space, the paper provides a mechanism to recover substantive reasoning steps that were previously invisible to selection algorithms.
• Scalability: The gains scale with problem difficulty; harder problems with longer evidence chains benefit most from the evidence discovery mechanism.

The authors maintain a modest stance, noting that the decision-evidence structure has been verified primarily in math and code, and that the current token classification relies on surface-level heuristics rather than causal intervention. They suggest future work could explore adaptive scheduling of the selection ratios ($p, q$) as training progresses.