Technical Summary: Anchored Residual Guidance for Privileged On-Policy Distillation

1. Problem Statement

The paper addresses a specific failure mode in Privileged On-Policy Distillation (Privileged OPD). In standard OPD, a student model learns from a teacher's predictions on states generated by the student itself, mitigating the exposure bias of supervised fine-tuning (SFT). Privileged OPD enhances this by using a "self-teacher" (the same base model) augmented with privileged information (e.g., oracle traces, gold solutions) to provide dense, answer-directed supervision.

However, the authors identify that treating this fully privileged teacher as an absolute imitation target creates a "reachability mismatch." Because the full-view teacher conditions on the complete future context (including the final answer), its probability distribution incorporates hindsight-biased information. This shifts probability mass toward tokens that are factually correct given the future but are causally unsupported by the student's current prefix.

• The Consequence: The student is incentivized to skip valid intermediate reasoning steps in favor of "locally unsupported shortcuts" to match the hindsight-biased distribution.
• The Pathology: This leads to privileged-information leakage, where the model appears to "cheat" by anticipating answers it cannot logically derive from its current state, a phenomenon that worsens as trajectory length increases (long-horizon drift).

2. Methodology: Anchored Residual On-Policy Distillation (AR-OPD)

To resolve the mismatch between the student's local predictive support and the teacher's hindsight-biased distribution, the authors propose AR-OPD, a dual-view framework that disentangles privileged supervision into two components: a Locally Compatible Anchor and a Destination-Directed Residual.

Core Mechanism

Instead of forcing the student to imitate the full privileged teacher ($q_{full}$), AR-OPD constructs a target distribution ($q_\lambda$) using the following decomposition in log-probability space:

$$\log q_\lambda(v | s_t) = \underbrace{\log q_{part}(v | s_t)}_{\text{Anchor}} + \lambda \underbrace{(\log q_{full}(v | s_t) - \log q_{part}(v | s_t))}_{\text{Residual}} - C_t$$

1. The Anchor ($q_{part}$): A partially privileged teacher is constructed by deterministically truncating the privileged context (e.g., keeping the first 50% of the reasoning trace, $\rho=0.5$). This teacher provides a causally reachable target that isolates the student from hindsight-biased answers while maintaining structural guidance.
2. The Residual: The difference between the full-view teacher and the partial-view teacher represents the marginal foresight (the "destination-directed" signal).
3. The Control Parameter ($\lambda$): This coefficient scales the residual.
   • If $\lambda = 0$, the method reduces to partial privileged OPD.
   • If $\lambda = 1$, it recovers the full privileged target.
   • The authors find that $\lambda < 1$ (contractive regime) is optimal, keeping the target anchored between the partial and full views to prevent over-transfer of hindsight information.

Training Objective

The student is trained via forward KL divergence to minimize the distance between its policy and this anchored residual target $q_\lambda$ on student-visited states, rather than the monolithic full-view target.

3. Key Contributions

1. Diagnosis of Hindsight Leakage: The paper provides mechanistic evidence that monolithic privileged OPD induces optimization pathologies by pulling students toward tokens outside their local predictive support, particularly in long-horizon trajectories (>768 tokens).
2. AR-OPD Framework: The introduction of a dual-view distillation method that operationalizes the separation of local reachability (Anchor) and future guidance (Residual).
3. Empirical Validation: Demonstration that controlling the full-view residual improves performance over both standard SFT and full privileged OPD, specifically by reducing shortcut events and mitigating late-stage drift.

4. Experimental Results

The authors evaluated AR-OPD across four domains: Mathematics (MATH500, AMC23), Code (HumanEval, MBPP), Science (SciKnowEval), and Medical QA (MedQA, MedCQA), using Qwen2.5-7B-Instruct as the base model.

• Performance Gains: AR-OPD achieved the highest average score (70.3) across all benchmarks.
  • +12.1 points over the base model.
  • +7.9 points over Supervised Fine-Tuning (SFT).
  • +2.3 points over the strong Full Privileged OPD baseline.
• Domain Specifics: Gains were most pronounced in mathematical reasoning, where AR-OPD reached 74.6 on MATH500 (exceeding Full OPD by 3.6 points).
• Long-Horizon Robustness: On trajectories exceeding 768 tokens, AR-OPD outperformed Full OPD by 7.2 points, demonstrating superior stability in extended reasoning tasks.
• Mechanism Verification:
  • Shortcut Reduction: AR-OPD reduced "shortcut events" (instances of privileged-information leakage) by 21.7% compared to Full OPD.
  • Signal Reliability: Diagnostics showed that the partial privileged view retains nearly as much predictive power for final correctness (ROC-AUC 0.766) as the full view (0.779), while the marginal full-minus-partial signal is significantly weaker (0.585), justifying the residual approach.

5. Significance and Claims

The paper claims that the utility of privileged information in distillation is not a binary choice (use vs. discard) but a target-design problem.

• Reframing Privilege: The authors argue that full-view oracle distributions contain useful destination-directed signals but are dangerous when treated as absolute targets due to hindsight bias.
• Controlled Guidance: By anchoring the target to a locally reachable state and injecting foresight as a controlled residual, AR-OPD allows models to benefit from "seeing more" without sacrificing the causal integrity of the reasoning process.
• Scalability: The method is particularly significant for long-horizon reasoning, where the reliability of full-view targets degrades, and the risk of hindsight-induced drift is highest.

The authors conclude that AR-OPD offers a robust recipe for post-training reasoning models, balancing the density of privileged supervision with the local compatibility required for stable on-policy learning.