Cross-Tokenizer LLM Distillation through a Byte-Level Interface

This paper introduces Byte-Level Distillation (BLD), a simple yet effective method that enables cross-tokenizer knowledge transfer by converting teacher outputs to byte-level probabilities and distilling them through a shared byte-level interface, achieving competitive or superior performance against more complex approaches across various model sizes and benchmarks.

The Gist

Imagine you are trying to teach a brilliant, world-class chef (the Teacher) how to cook a specific dish, but you want the student to learn from them even though they speak completely different languages.

• The Teacher speaks "Token-ese." They chop ingredients into specific chunks called "tokens" (like "chicken," "breast," "sliced").
• The Student speaks "Byte-ese." They chop ingredients into tiny, individual atoms called "bytes" (like "c," "h," "i," "c," "k," "e," "n").

In the world of AI, this is a huge problem. Usually, to teach a student, you need them to speak the exact same language as the teacher. If the teacher says "chicken" (one token) and the student only understands "c-h-i-c-k-e-n" (seven bytes), they can't compare notes. The teacher's instructions get lost in translation.

The Old Way: Trying to Force a Translation

Previous methods tried to solve this by building complex dictionaries or guessing how to map the teacher's "chunks" to the student's "atoms." It's like trying to translate a poem by guessing which word in the new language sounds like the old one. It's messy, prone to errors, and often loses the nuance of the original meaning.

The New Idea: The "Byte-Level" Universal Translator

This paper introduces a clever new method called Byte-Level Distillation (BLD). Instead of trying to translate the words (tokens), they decided to translate the letters (bytes).

Here is the analogy:
Imagine the Teacher and Student are both trying to describe a picture of a cat.

• The Teacher describes it as: "Cat," "Fluffy," "Orange." (Tokens)
• The Student describes it as: "C," "a," "t," "F," "l," "u," "f," "f," "y"... (Bytes)

The researchers realized that every single word in every language is made of the same 256 tiny building blocks (bytes). Whether you are speaking English, Chinese, or code, the underlying "atoms" are the same.

How BLD works (The 3-Step Recipe):

1. The Teacher's Secret Decoder: The researchers take the Teacher's output (the "Token" words) and mathematically break them down into their "Byte" probabilities. Instead of saying "There is a 90% chance the next word is 'Cat'," the Teacher now says, "There is a 90% chance the next letter is 'C', then 'a', then 't'."
2. The Student's New Goggles: They give the Student a special, lightweight pair of "Byte Goggles" (a small extra brain module). This allows the Student to look at the Teacher's "Byte" instructions and learn from them directly.
3. The Lesson: The Student learns to predict the next byte based on the Teacher's byte-by-byte guidance. Once the lesson is over, they take the "Byte Goggles" off. The Student is now a normal AI, but it has learned the Teacher's wisdom without ever needing to speak the Teacher's specific "Token" language.

Why is this cool?

• It's Simple: You don't need complex dictionaries or messy mappings. You just go down to the smallest common denominator: the byte.
• It Works: In their tests, this simple method worked just as well as, and sometimes better than, very complicated methods that tried to force the vocabularies to match.
• It's Flexible: You can teach a model trained on medical jargon to a model trained on legal jargon, even if they use completely different ways of chopping up words.

The Catch (The "Sobering" Reality)

The paper ends with a very honest note. While this new method is great, it's not a magic wand that fixes everything.

• Sometimes the Student learns better at math but gets worse at following instructions.
• Sometimes the Student learns better at one type of task but fails at another.

The Big Takeaway:
The researchers found that while "Byte-Level Distillation" is a fantastic new tool in the toolbox, Cross-Tokenizer Distillation (teaching AI models with different vocabularies) is still a giant, unsolved puzzle. We have found a better way to start the conversation, but we haven't figured out how to make the student perfectly mimic the master in every single situation yet.

In short: They found a universal language (bytes) that lets different AI models talk to each other without needing a translator, but the conversation is still a work in progress.

Technical Summary

1. Problem Statement

Cross-Tokenization Distillation (CTD) is the challenge of transferring knowledge from a teacher Large Language Model (LLM) to a student LLM when they utilize different tokenizers and, consequently, different vocabularies.

• The Core Bottleneck: Standard knowledge distillation relies on matching the probability distributions (logits) of the teacher and student over a shared vocabulary. If the tokenizers differ (e.g., one uses BPE, another uses Byte-Pair Encoding, or one is byte-level), their output spaces are incompatible. A logit vector of size 50,000 cannot be directly compared to one of size 32,000.
• Limitations of Existing Solutions: Current approaches attempt to bridge this gap using heuristic strategies, such as:
  • Distilling from generated text samples (inefficient, high information loss).
  • Creating ad-hoc mappings between vocabularies or hidden states (computationally expensive, lack theoretical grounding).
  • Approximate likelihood matching (introduces complexity and alignment errors).
• The Goal: To establish a principled, efficient, and alignment-free method for CTD that allows models with heterogeneous tokenizers to share knowledge effectively.

2. Methodology: Byte-Level Distillation (BLD)

The authors propose Byte-Level Distillation (BLD), a method that bypasses vocabulary mismatch by operating at the byte level, a representation common to all tokenizers (since all tokens are ultimately composed of bytes).

The method consists of two primary steps:

Step 1: Byte-Level Interface Construction

• Teacher Side: The teacher's token-level output distribution is converted into byte-level probabilities. This is achieved using a fast approximation algorithm (based on Vieira et al., 2025) that sums the probabilities of all token sequences that "cover" a specific byte sequence. This allows the calculation of $P_T(b_i | b_{<i})$, the probability of the $i$-th byte given previous bytes.
• Student Side: A lightweight, learnable byte-level decoder head ($O_b$) is added in parallel to the student's existing token-level output layer.
  • The student's transformer backbone remains largely unchanged (often fine-tuned via LoRA).
  • The new head maps hidden states to a probability distribution over the byte alphabet ($\Sigma$, size 256 + special tokens).
  • To manage computational load, the head is often designed to predict a fixed number of bytes (e.g., 10) per token position.

Step 2: Distillation Process

The training objective combines three loss components to update the student:

1. Next-Token Cross-Entropy (CE): Standard supervised loss on the student's token-level head.
2. Next-Byte Cross-Entropy (CE): Supervised loss on the new byte-level head, ensuring the student learns to predict bytes correctly.
3. Byte-Level KL Divergence: The core distillation term. It minimizes the Kullback-Leibler divergence between the teacher's byte-level probability distribution and the student's byte-level predictions.

Post-Training: Once distillation is complete, the byte-level decoder head is removed, leaving a standard token-level model that has inherited the teacher's knowledge despite having a different tokenizer.

3. Key Contributions

1. A Simple, Alignment-Free Baseline: The paper introduces BLD, which eliminates the need for complex vocabulary mapping or heuristic alignment strategies by utilizing the universal byte interface.
2. Competitive Performance: Empirical results show that BLD performs competitively with, and often surpasses, significantly more sophisticated state-of-the-art CTD methods (like Universal Logit Distillation or Dual-Space Distillation) across various tasks.
3. Critical Insight on CTD: The authors demonstrate that while BLD is effective, no single method (including BLD) achieves consistent gains across all benchmarks. This highlights that CTD remains a fundamentally open and challenging problem.

4. Experimental Results

The authors evaluated BLD on three tasks using models ranging from 1B to 8B parameters:

• Task A: BPE-to-BPE Tokenizer Transfer

  • Setup: Transferring Llama 3.2 3B to the Qwen2 tokenizer.
  • Results: BLD achieved the highest scores on PiQA (75.68) and AGI-ZH (35.97), recovering performance close to the original model on MMLU and BoolQ.
  • Weakness: BLD lagged significantly in Instruction Following (IFEval), suggesting byte-level distillation may struggle to preserve structured output behaviors required for complex instructions.

• Task B: BPE-to-Byte Tokenizer Transfer

  • Setup: Adapting Llama 3.2 3B to a pure byte-level tokenizer.
  • Results: This task proved much harder, with all methods suffering significant performance drops (e.g., ~21 points on MMLU). BLD ranked first on PiQA but showed no clear dominance across other benchmarks, indicating that byte-level transfer is still an unsolved challenge.

• Task C: Cross-Model Distillation

  • Setup: Distilling a math-specialized 8B model (OpenMath2-Llama3.1) into a 2B model (Gemma2).
  • Results: BLD achieved the highest GSM8K score (62.55), outperforming SFT and ALM+SFT. However, it underperformed SFT on the MATH benchmark.
  • Observation: The gap between the distilled student and the teacher remained large, confirming that effective cross-tokenizer transfer across heterogeneous models is difficult.

5. Significance and Future Directions

• Practical Implications: BLD enables the creation of specialized expert models (e.g., for medicine or law) by distilling general-purpose models into architectures with domain-specific tokenizers, without needing to retrain from scratch. It also facilitates ensembling knowledge from multiple heterogeneous open-source models.
• Theoretical Insight: The paper establishes the byte level as a natural "common ground" for knowledge transfer, validating that token-level probability distributions can be effectively projected down to bytes for distillation purposes.
• Open Problem: The most significant finding is the lack of a "silver bullet." The performance of distillation methods is highly dependent on the specific benchmark, model pair, and task type. This underscores that CTD is not yet solved and requires further investigation into how to better preserve specific capabilities (like instruction following or complex reasoning) during cross-tokenizer transfer.

In summary, Byte-Level Distillation offers a simple, effective, and theoretically grounded baseline for cross-tokenizer knowledge transfer, successfully bypassing vocabulary mismatches, though it reveals that the broader challenge of consistent cross-model distillation remains open.