Technical Summary: SimCT - Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation

1. Problem Statement

On-policy distillation (OPD) is a standard paradigm for transferring the behavior of a large teacher model to a smaller student model. Unlike offline distillation, OPD queries the teacher on prefixes generated by the student's current policy, thereby aligning the training distribution with the inference distribution and mitigating error compounding.

However, standard OPD implicitly assumes that the teacher and student models share a tokenizer, allowing for token-by-token comparison of their next-token predictions. In realistic deployment scenarios, teacher and student models often belong to different families with heterogeneous tokenizers. This mismatch creates two fundamental obstacles:

1. Vocabulary Mismatch: The sets of tokens ($\mathcal{V}_T$ and $\mathcal{V}_S$) overlap only partially. Restricting supervision to shared tokens discards a significant portion of the teacher's output distribution.
2. Sequence Mismatch: Even when the underlying text is identical, different tokenizers segment it into different token sequences (e.g., "happy" vs. "hap" + "py" vs. "ha" + "pp" + "y"). Consequently, the teacher's next-token distribution at a specific prefix may not correspond to the same text unit as the student's distribution, rendering standard token-level KL divergence ill-defined.

Existing cross-tokenizer solutions attempt to bridge this gap via optimal transport (computationally expensive), learned projection layers (requiring extra parameters), or coarse-grained chunk matching (losing fine-grained signal). These methods often fail to recover the full teacher signal or introduce significant overhead.

2. Methodology: Simple Cross-Tokenizer OPD (SimCT)

The authors propose SimCT, a method that restores lost supervision by expanding the supervision space rather than modifying the OPD objective function itself.

Core Concept: Minimal Aligned Units

Instead of forcing predictions to match only on identical tokens, SimCT identifies Minimal Aligned Units (MAUs). An MAU is defined as the finest text continuation that both tokenizers can realize, even if they decompose it into different token sequences.

• Shared Tokens: Tokens present in both vocabularies ($\mathcal{V}_T \cap \mathcal{V}_S$) are treated as 1:1 aligned units.
• MAUs: For text regions where tokenizers disagree, the method constructs the finest boundary-consistent partition where each unit can be represented as a sequence of teacher tokens and a sequence of student tokens.

Supervision Space Construction

SimCT defines a common supervision space $\mathcal{U}_{SimCT}$:
$$\mathcal{U}_{SimCT} = (\mathcal{V}_T \cap \mathcal{V}_S) \cup \mathcal{A}$$
where $\mathcal{A}$ is the set of all minimal aligned units.

Scoring and Loss

To compare distributions over this space, SimCT introduces scoring maps $\Pi_T$ and $\Pi_S$:

1. Shared Tokens: The score is the original log-probability.
2. MAUs (Multi-token): The score is the length-normalized average log-likelihood of the constituent tokens. For a unit $u$ realized as $k$ tokens $(v_1, \dots, v_k)$ by model $M$:
   $$s_M(u | x_{<t}) = \frac{1}{k} \sum_{j=1}^k \log p_M(v_j | x_{<t}, v_{<j})$$
   This normalization prevents penalizing units that require more tokens under a specific tokenizer.

The final distributions $q_T$ and $q_S$ over $\mathcal{U}_{SimCT}$ are obtained by normalizing these scores. The distillation loss remains the standard reverse-KL divergence:
$$\mathcal{L}_{SimCT} = \text{KL}(q_S(\cdot | x_{<t}) \parallel q_T(\cdot | x_{<t}))$$

Theoretical Properties

• Theorem 1: MAUs constitute the finest boundary-consistent interface jointly expressible by both tokenizers. No strictly finer decomposition preserves joint tokenizability.
• Proposition 1: Coarsening these units (merging MAUs into larger spans) strictly reduces the KL divergence between teacher and student distributions, effectively erasing useful within-span distinctions. Thus, preserving the minimal granularity is theoretically optimal for retaining signal.

3. Key Contributions

1. Identification of Signal Loss: The paper demonstrates that standard cross-tokenizer OPD silently discards a large fraction of teacher signal due to both vocabulary and sequence mismatches.
2. Supervision Space Expansion: SimCT introduces a novel supervision interface that recovers signal from non-shared vocabulary and non-1:1 aligned text segments without requiring additional trainable parameters or complex transport computations.
3. Theoretical Justification: The authors prove that MAUs represent the finest possible supervision granularity and that coarsening them inevitably removes useful teacher-student distinctions.
4. Empirical Validation: Extensive experiments show that SimCT consistently outperforms shared-vocabulary baselines and other cross-tokenizer methods across mathematical reasoning and code generation tasks.

4. Experimental Results

The authors evaluated SimCT on three heterogeneous teacher-student pairs (Qwen2.5, Phi-4, Gemma-2) across four benchmarks: GSM8K, MATH-500, MBPP, and LiveCodeBench-v6.

• Performance Gains: SimCT achieved consistent improvements over SFT warm-starts and all baselines (SimpleOPD, DSKD, ALM, GOLD). It yielded an average relative gain of 6.0% over SFT and outperformed the best cross-tokenizer baseline by a significant margin.
• Signal Recovery Analysis: Ablation studies confirmed that the gains stem from recovering two types of discarded signals:
  • Out-of-Shared-Vocab (OoSV): High-probability teacher predictions outside the shared vocabulary.
  • Unaligned Units: Text segments that cannot be aligned 1:1.
    Recovering both sources via the unified MAU space yielded the best performance.
• Granularity Ablation: Merging MAUs into coarser units resulted in a drop in performance, correlating with the theoretical prediction that coarsening removes the KL signal representing fine-grained teacher-student preferences.
• Efficiency: SimCT adds only a marginal computational overhead compared to SimpleOPD (approx. 0.16 hours per run on 8x H20 GPUs) and is significantly faster than methods requiring learned projections or optimal transport.

5. Significance and Claims

The paper argues that the primary bottleneck in cross-tokenizer OPD is not the training paradigm itself, but the construction of the supervision space. By shifting the focus from "matching tokens" to "matching text continuations," SimCT recovers supervision that was previously inaccessible.

The authors claim that SimCT provides a simple, parameter-free, and robust solution for distilling between models with different tokenizers. The results suggest that preserving the finest jointly tokenizable units is critical for maintaining the quality of on-policy learning signals. The work establishes that expanding the supervision space to include minimal aligned units is a more effective strategy than modifying the loss function or adding complex alignment machinery.

Limitations noted by the authors:

• The evaluation is currently limited to math and code domains.
• The method relies on standard BPE-style tokenizers; adaptation may be needed for tokenizers with fundamentally different segmentation behaviors.
• The current implementation assumes white-box access to teacher next-token distributions; extension to black-box settings is left for future work.