TiTok: Transfer Token-level Knowledge via Contrastive Excess to Transplant LoRA

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed in real-world scenarios, but full fine-tuning is computationally expensive. Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA (Low-Rank Adaptation) mitigate this by updating only a small subset of parameters. However, a critical limitation exists: LoRA adapters are tightly coupled to their specific base model and cannot be directly transferred to different backbones (e.g., from Mistral to Llama).

Existing solutions face significant drawbacks:

• Knowledge Distillation (KD): Typically requires access to the original downstream training data, which is often unavailable or proprietary.
• TransLoRA: A recent approach that uses synthetic data to transfer LoRA knowledge. However, it requires training an additional discriminator model to filter low-quality synthetic data, introducing extra complexity, computational overhead, and pipeline complexity.

The core challenge is to enable LoRA transplantation across different models (different families, sizes, or versions) without original training data, without auxiliary discriminator models, and with minimal computational overhead.

2. Methodology: TITOK Framework

The authors propose TITOK (Transfer Token-Level Knowledge via Contrastive Excess), a framework that transfers knowledge by selectively training on informative tokens within synthetic data. The process consists of four main stages:

A. Synthetic Data Generation

Instead of using the target model to generate queries (as in TransLoRA), TITOK uses the Source Expert Model ($M_s + A_s$, where $M_s$ is the base and $A_s$ is the LoRA adapter) to generate both queries ($q$) and labels ($y$) based on a few-shot seed prompt. This ensures the synthetic data aligns closely with the source adapter's knowledge distribution.

B. Token-Wise Contrastive Excess Score

This is the core innovation. TITOK computes an "excess score" for every token in the generated response to identify where the LoRA adapter injects specific task knowledge.

• Amateur Model: The source base model alone ($M_s$).
• Expert Model: The source base model with LoRA ($M_s + A_s$).
• Calculation: For a token $y_i$ in response $y$ given query $q$, the excess score $S(y_i)$ is defined as the difference in log-probabilities (losses) between the expert and amateur models:
  $$S(y_i) = \log P_{M_s+A_s}(y_i | q, y_{<i}) - \log P_{M_s}(y_i | q, y_{<i})$$
• Interpretation: A high positive score indicates that the LoRA adapter significantly increased the confidence of predicting that token compared to the base model. These tokens represent task-specific knowledge that the backbone lacks. This metric is theoretically grounded in the Likelihood Ratio Test (LLR), identifying regions where the two distributions diverge most.

C. Two-Stage Filtering

TITOK filters the synthetic dataset ($D_s$) using the excess scores before training the target model:

1. Sample Filtering: Computes the mean excess score for each sample. Only the top $M$ samples with the highest average scores are retained. This removes uninformative synthetic examples.
2. Token Selection: Within the retained samples, only the top $k\%$ of tokens (ranked by their individual excess scores) are used for training. The target model is trained only on these high-value tokens, ignoring the rest.

D. Tokenizer Alignment

When transferring between models with different tokenizers (e.g., Mistral vs. Llama), direct token mapping is impossible. TITOK employs a dual-pointer alignment algorithm:

• It aligns the source and target token sequences by incrementally decoding text spans.
• It propagates the binary masks (keep/discard) from source to target using four rules:
  1. One-to-One: Direct copy.
  2. One-to-Many: Replicate the score.
  3. Many-to-One: Average the scores.
  4. Many-to-Many: Average and replicate.
• This allows the target model to learn from the correct semantic regions even if the tokenization differs.

3. Key Contributions

1. Token-Level Knowledge Transfer: Unlike previous methods that transfer entire sequences or rely on global logit alignment, TITOK focuses on token-level signals, identifying and transferring only the specific tokens where the adapter's knowledge is most critical.
2. Discriminator-Free Design: TITOK eliminates the need for an auxiliary discriminator model. It uses the source model itself (as both "amateur" and "expert") to generate supervision signals, significantly reducing computational overhead and pipeline complexity.
3. Robust Tokenizer Alignment: The proposed alignment algorithm enables effective transfer across models with different tokenizers, a common hurdle in cross-architecture LoRA transplantation.
4. Data Efficiency: The method works effectively with synthetic data generated from few-shot prompts and can even utilize external data from different domains/users, demonstrating high flexibility.

4. Experimental Results

The authors evaluated TITOK on three benchmarks: Big-Bench Hard (BBH), MMLU (reasoning), and LaMP (personalization/generation).

• Performance Gains:
  • TITOK consistently outperformed baselines (Vanilla, KD, and TransLoRA) across all transfer settings.
  • Average Improvements: +9.94% over Vanilla, +8.5% over KD, and +4.4% over TransLoRA.
  • Cross-Model Transfer: Effective in transferring from Mistral to Llama, small models to large models (3B $\to$ 8B), and different versions (Llama-2 $\to$ Llama-3).
• Ablation Studies:
  • Both Sample Filtering and Token Selection were shown to be essential; removing either degraded performance.
  • Using the Source Expert for query generation yielded better results than using the Target model.
• Robustness:
  • TITOK remained effective even when using low-quality synthetic data or external data from different users/tasks.
  • It showed robustness to different prompt designs and selection ratios ($k\%$).
• Efficiency:
  • TITOK achieved a 1.5x–2.5x reduction in total compute time compared to TransLoRA, primarily due to the elimination of discriminator training.

5. Significance

TITOK represents a paradigm shift in PEFT knowledge transfer. By moving from sequence-level or logit-level distillation to token-level selective transfer, it addresses the "noise" inherent in synthetic data and the rigidity of current LoRA transplantation methods.

• Practicality: It offers a lightweight, plug-and-play solution for deploying LoRA adapters across diverse model families without retraining from scratch or accessing proprietary data.
• Scalability: The method scales well to larger models (e.g., 70B parameters) and complex architectures (MoE).
• Generalizability: Its ability to function with external data and different tokenizers makes it highly applicable to real-world scenarios where data privacy and model heterogeneity are major constraints.

In summary, TITOK provides a simple yet powerful mechanism to "transplant" specialized knowledge from one model to another by focusing on the precise tokens that carry that knowledge, bypassing the need for heavy auxiliary models or original datasets.