Technical Summary: On the Geometry of On-Policy Distillation

Problem Statement

Large Reasoning Models (LRMs) have advanced significantly through post-training paradigms, primarily Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR). Recently, On-Policy Distillation (OPD) has emerged as a complementary approach, training a student model on its own sampled trajectories using dense token-level guidance from a stronger teacher. While OPD demonstrates empirical utility, its parameter-space dynamics remain poorly understood. Specifically, it is unclear whether OPD represents a simple interpolation between the geometric footprints of SFT and RLVR, or if it induces a distinct optimization trajectory. The paper addresses three core questions:

1. Where does OPD lie within the parameter-space spectrum between SFT and RLVR?
2. What intrinsic update trajectory does OPD follow during training?
3. Which component of the OPD objective controls this trajectory?

Methodology

The authors analyze the geometry of OPD updates using a suite of parameter-space diagnostics applied to checkpoints of the Qwen3 family (specifically Qwen3-8B). The experimental setup compares three paradigms initialized from a shared SFT anchor (for OPD and RLVR) or a pretrained base (for SFT):

• SFT: Standard supervised fine-tuning on offline demonstrations.
• RLVR: Policy optimization using sparse outcome-level rewards (GRPO).
• OPD: On-policy distillation using dense token-level teacher supervision.

The analysis employs four primary diagnostics to characterize updates ($\Delta W = W_+ - W_0$):

1. Update Sparsity: Measuring the fraction of weights unchanged at bfloat16 precision.
2. Principal-Angle Rotation: Quantifying the rotation of dominant singular subspaces ($U, V$) between initial and final weights.
3. Spectral Drift: Measuring the normalized spectral shift (NSS) of singular values.
4. Update-Mask Overlap: Determining whether visible updates concentrate on principal (high-curvature) or low-magnitude weights.

To investigate training dynamics, the authors track cumulative updates ($\Delta W_t$) across checkpoints using stable rank, Frobenius norms, and Hill tail estimates. They further conduct control experiments perturbing three factors: token supervision density (sparsification), rollout policy (off-policy generation), and objective composition (interpolating OPD and RLVR advantages).

Key Contributions and Results

1. OPD Occupies a "Relaxed Off-Principal" Regime

The study locates OPD in a distinct region of the parameter space, intermediate between SFT and RLVR but biased toward the RLVR side.

• Sparsity: OPD exhibits intermediate update sparsity (51.6% unchanged weights), falling between the dense updates of SFT (8.1% unchanged) and the sparse updates of RLVR (77.2% unchanged).
• Geometry: OPD induces less subspace rotation and spectral drift than SFT but more than RLVR.
• Localization: OPD updates shift away from principal weights and toward low-magnitude regions, similar to RLVR, but with a larger visible support (lower selectivity).
• Interpretation: The authors propose a "Relaxed Three-Gate" view. OPD retains the geometry-preserving bias of RLVR (Gate II: model geometry steering updates away from principal directions) but relaxes the selectivity due to dense token-level supervision (Gate I: distributional anchor), allowing more coordinates to cross the precision realization threshold (Gate III).

2. Subspace Locking

OPD follows a distinct trajectory characterized by subspace locking.

• Early Convergence: Unlike SFT (which expands its update subspace) or RLVR (which contracts late), OPD rapidly enters a narrow, low-dimensional update band early in training.
• Stability: This low-dimensional channel emerges early (within the first measured checkpoints) and remains persistent throughout training.
• Functional Sufficiency: Constraining subsequent training to the low-dimensional subspace (rank-16) formed early in OPD training preserves OPD performance. In contrast, the same constraint significantly degrades SFT performance. This indicates the locked subspace is functionally sufficient for OPD learning.
• Magnitude: OPD accumulates a substantially larger cumulative update norm than RLVR while maintaining a comparable stable rank, ruling out the possibility that low dimensionality is merely an artifact of negligible parameter movement.

3. Objective Composition Controls the Trajectory

Control experiments reveal that the subspace locking mechanism is robust to runtime perturbations but sensitive to objective composition.

• Robustness: Token sparsification (reducing supervision density) and off-policy rollouts (using teacher-generated data) preserve the OPD stable-rank trajectory, affecting only the update scale.
• Sensitivity: Interpolating the OPD objective with the RLVR advantage signal alters the rank dynamics. When the OPD component is weakened, the trajectory departs from the locked regime.
• Mechanism: The authors argue that token sparsification and off-policy rollouts merely rescale the gradient covariance, preserving the dominant spectral directions. In contrast, mixing objectives changes the gradient source itself, altering the dominant covariance geometry and breaking the locked trajectory.

Significance and Claims

The paper claims that OPD is not merely an intermediate endpoint between SFT and RLVR but induces its own unique update geometry. The primary significance lies in identifying subspace locking as a fundamental property of OPD training.

The authors suggest that future OPD algorithms should be designed with geometry control in mind rather than treating token-level supervision density as the primary design axis. Effective recipes should:

• Monitor and exploit the early-emerging, persistent low-dimensional update channel.
• Use objective composition as the primary lever for regulating update geometry.
• Treat token selection and rollout policies as secondary factors that influence update scale rather than the fundamental spectral structure.

The paper concludes that these diagnostics provide a parameter-space account of OPD, offering a foundation for more stable, interpretable, and transferable geometry-aware optimization strategies. The authors note limitations regarding the focus on controlled Qwen3 reasoning settings and the reliance on checkpoint-based diagnostics rather than formal causal theories.