Technical Summary: Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families

1. Problem Statement

On-Policy Distillation (OPD) has emerged as a highly efficient technique for post-training Large Language Models (LLMs), enabling the transfer of knowledge from expert "teacher" models to "student" models by leveraging dense, token-level supervision signals (log-probabilities) on the student's own generated trajectories. However, existing OPD methods are strictly confined to same-family distillation, where the teacher and student must share an identical tokenizer.

When attempting cross-tokenizer distillation (e.g., between different model families like Qwen and Llama), a fundamental barrier arises: the misalignment of vocabularies. Differences in vocabulary size, encoding preferences, and the handling of special characters mean that raw log-probabilities from the teacher cannot be directly mapped to student tokens. Current mainstream practices for cross-tokenizer scenarios resort to Supervised Fine-Tuning (SFT) on teacher-generated responses. This approach fails to capture the rich, dense knowledge embedded in the teacher's probability distribution, treating the teacher's output merely as static text rather than a probabilistic signal. Furthermore, existing cross-tokenizer alignment methods are primarily designed for offline (off-policy) distillation and cannot effectively solve the token-level signal propagation problem inherent in OPD.

2. Methodology

The authors propose a framework for Cross-Tokenier On-Policy Distillation that bridges the vocabulary gap without requiring shared tokenizers or modifying the models' internal architectures. The method consists of two core components:

A. Adaptive Chunk Alignment (Dual-Pointer Chunk Alignment - DPCA)

To align sequences generated by different tokenizers, the authors introduce a deterministic algorithm that identifies minimal synchronized chunks.

• Mechanism: The algorithm utilizes the deterministic and monotonic properties of the detokenization function ($D$). It employs a dual-pointer strategy ($p_S$ for student, $p_T$ for teacher) to find the smallest increments of tokens such that the decoded text prefixes match: $D(y^S_{1:i}) = D(y^T_{1:j})$.
• Greedy Strategy: The pointers advance based on the length of the decoded strings. If the student's decoded fragment is shorter than the teacher's, the student pointer advances, and vice versa.
• Theoretical Guarantee: The algorithm ensures completeness (identifying all synchronization points) and minimality (ensuring chunks cannot be subdivided further). This guarantees that the dense supervision signals are localized to the smallest possible token clusters, avoiding signal ambiguity over large, aggregated blocks.

B. Chunk-Level Credit Assignment via Semantic Prior

Once sequences are partitioned into synchronized chunks, the teacher's supervision signals must be projected onto the student's tokens.

• Joint Probability Matching: The method assumes the joint probability of generating a synchronized chunk (which represents the same underlying text) should be equivalent under both distributions. This implies the sum of log-probabilities within a chunk must be equal.
• Optimization with Semantic Prior: To distribute the teacher's target log-likelihood ($L^{(c)}_T$) across the student's tokens ($s_c$) while preserving the student's internal semantic structure, the authors solve a constrained optimization problem. They seek a target distribution that maintains maximum directional alignment with the student's prior distribution ($p$).
• Closed-Form Solution: The resulting target log-probabilities ($\log q_i$) are a linearly scaled version of the student's prior log-probabilities ($\log p_i$), scaled by the ratio of the chunk-level log-likelihoods:
  $$\log q_i = \frac{L^{(c)}_T}{L^{(c)}_S} \log p_i$$
  This ensures that tokens the student initially found "surprising" (lower probability) receive a proportionally larger share of the advantage signal, preserving the relative semantic structure within the chunk.

C. Training Objective

The calculated per-token advantages are integrated into a standard OPD framework using an RL importance-sampling loss function (Reverse KL divergence), allowing the student to update its policy based on its own rollouts while adhering to the teacher's guidance.

3. Key Contributions

1. First High-Fidelity Cross-Tokenier OPD: This work is the first to enable high-fidelity, per-token on-policy distillation across different model families with distinct tokenizers.
2. Adaptive Alignment Algorithm: The proposal of the Dual-Pointer Chunk Alignment (DPCA) algorithm, which mathematically guarantees the identification of minimal synchronized chunks between heterogeneous tokenization schemes.
3. Semantic-Prior Credit Assignment: A novel mechanism to project teacher log-probabilities onto student tokens that preserves the student's internal semantic structure, avoiding the distortion of relative token probabilities.
4. Compute Efficiency: Demonstrating that cross-tokenizer OPD is significantly more compute-efficient than traditional SFT or off-policy distillation baselines.

4. Experimental Results

The authors evaluated their method on reasoning benchmarks (AIME, MATH, GPQA-Diamond, LiveCodeBench) using two distinct teacher-student pairings:

• Setting 1: Qwen3-8B (Teacher) $\to$ Llama-3.1-8B-SFT (Student).
• Setting 2: DeepSeek-R1 (Teacher) $\to$ DeepSeek-R1-Distill-Qwen-7B (Student).

Performance:

• In the Qwen $\to$ Llama setting, OPD achieved an average score of 40.7, outperforming SFT (35.1), ALM (35.7), and CDM (35.9) by significant margins (up to +5.6 points).
• In the DeepSeek-R1 setting, OPD reached 52.0, outperforming baselines by 2.4–2.7 points on average.
• Gains were consistent across math, science, and code benchmarks, with particularly notable improvements on AIME25 (+11.6 points) and GPQA-Diamond (+6.1 points).

Compute Efficiency:

• To achieve the same AIME24 accuracy (44.4%) starting from an SFT-initialized model, the authors estimated that extending SFT would require 482K additional training samples and approximately 1.17 $\times$ 10$^{21}$ FLOPs.
• In contrast, the OPD method achieved the same performance with only 20K additional samples and approximately 4.8 $\times$ 10$^{19}$ FLOPs.
• This represents a reduction in training data usage to ~4.1% and total computation cost to ~4% of the SFT extrapolation scheme.

Comparison with Same-Tokenier OPD:

• The paper notes that the performance gain of cross-tokenizer OPD (average +5.6) is comparable to same-tokenizer OPD (average +5.5), indicating that the projection mechanism does not significantly degrade the quality of the supervision signal.

5. Significance and Claims

The paper claims that this work unlocks a broader range of teacher–student pairs for on-policy distillation. By removing the requirement for shared tokenizers, the method allows for more flexible knowledge transfer between diverse model families. The authors argue that a shared vocabulary is not a prerequisite for effective OPD, provided that teacher feedback can be projected onto student tokens in a log-probability-conserving manner.

The work opens new avenues for adapting and enhancing interactions between LLMs, suggesting that the "dense supervision" paradigm can be extended beyond single-model-series post-training. The authors acknowledge limitations regarding the scale of models tested (under 8B parameters due to computational constraints) and identify validation on larger-scale models (e.g., 70B) as future work.