LinguDistill: Recovering Linguistic Ability in Vision- Language Models via Selective Cross-Modal Distillation

1. Problem Statement

The paper addresses a critical issue in the development of Vision-Language Models (VLMs): the degradation of native linguistic capabilities when a pre-trained Language Model (LM) is adapted for multimodal tasks.

• The Phenomenon: When a frozen LM is fine-tuned with visual data (multimodal adaptation), it suffers from "representation shift" and "cross-modal interference." This leads to catastrophic forgetting of linguistic priors, resulting in significant performance drops on text-only reasoning and knowledge benchmarks (e.g., HellaSwag, ARC, ScienceQA), even if the model performs well on visual tasks.
• Limitations of Existing Solutions:
  • Standard Fine-tuning: Further fine-tuning on language-heavy data is insufficient to reverse the representation shift caused by multimodal training.
  • Adapter-based Approaches: Methods like BLIP-2 or LLaMA-Adapter preserve linguistic capabilities by freezing the LM and adding intermediate alignment modules. However, these increase architectural complexity, add parameters, and incur inference costs.
  • Parameter Manipulation: Techniques like weight interpolation or task arithmetic often require large-scale retraining or assume capabilities can be cleanly isolated in parameter space, which is not always feasible.

2. Methodology: LINGUDISTILL

The authors propose LINGUDISTILL, an adapter-free knowledge distillation framework that recovers linguistic ability without adding parameters or inference overhead. The core idea is to use the original frozen LM as a "teacher" to guide the multimodal "student" (VLM).

A. Key Challenge: Vision-Conditioned Supervision

A standard frozen LM cannot act as a teacher for a VLM because it only processes text. It lacks the visual context required to generate accurate supervision signals for multimodal generation.

B. Core Innovation: Layer-wise KV-Cache Sharing

To solve the alignment problem, LINGUDISTILL introduces a KV-cache sharing mechanism during training:

1. Architecture: The student VLM decoder ($\Phi$) and the frozen teacher LM decoder ($\Omega$) share the same architecture.
2. Process:
   • The student processes the full multimodal sequence (Image + Text) and generates Key-Value (KV) caches for all transformer layers.
   • The teacher processes only the text prompt to generate its Query (Q) states.
   • Crucial Step: The teacher directly reuses the student's KV caches ($K_\Phi, V_\Phi$) at every layer.
   • The teacher computes attention using its own queries ($Q_\Omega$) over the student's multimodal memory ($K_\Phi, V_\Phi$).
3. Result: This allows the frozen text-only LM to "see" the visual context and produce vision-aware supervision signals without modifying its weights or architecture. After training, the teacher is discarded.

C. Selective Distillation Objective

The authors recognize that a text-only teacher is not always reliable, especially for tasks requiring precise visual grounding (e.g., OCR, document layout). Therefore, they employ a data-dependent weighting strategy:

• Loss Function: A weighted combination of soft distillation loss (from the teacher) and hard cross-entropy loss (ground truth).
  $$L = \sum \alpha(d_b) L_{soft} + (1 - \alpha(d_b)) L_{hard}$$
• Selective Routing:
  • Language-heavy data (e.g., ScienceQA, ARC): High weight ($\alpha > 0$) is applied to the distillation loss to recover linguistic priors.
  • Vision-heavy/OCR data (e.g., DocVQA, OCRBench): The distillation signal is suppressed ($\alpha = 0$), relying solely on hard labels. This prevents the text-only teacher from misleading the student on pixel-level or layout-specific tasks.

3. Key Contributions

1. Adapter-Free Recovery: A novel method to restore linguistic capabilities in VLMs using the original frozen LM as a teacher, requiring zero additional parameters and no inference cost.
2. KV-Cache Sharing Architecture: A training-time mechanism that enables a frozen text-only model to attend to multimodal representations, effectively bridging the modality gap for distillation.
3. Selective Distillation Strategy: Demonstrates that uniform distillation harms visual tasks, whereas selectively applying distillation only to language-intensive data recovers linguistic performance while preserving visual grounding.

4. Experimental Results

The method was evaluated on the nanoVLM-460M model using The Cauldron dataset (50 VL instruction-tuning datasets).

• Linguistic Recovery: LINGUDISTILL recovered approximately 10% of the performance lost on language and knowledge benchmarks compared to standard fine-tuning.
  • ScienceQA: Improved from 0.592 (fine-tuned) to 0.676 (LINGUDISTILL), surpassing the original pre-trained baseline (0.590).
  • COCO Captioning: Recovered from 0.673 to 0.866.
  • HellaSwag: Reduced regression from -19.5% to only -2.7%.
• Visual Task Preservation:
  • Unlike uniform distillation (which caused a 40.5% drop on OCRBench), LINGUDISTILL maintained strong performance on visual tasks.
  • DocVQA: Only a 3.8% drop (vs. 16.8% drop with uniform KD).
  • OCRBench: A 21.1% drop (vs. 40.5% with uniform KD), showing that suppressing distillation on OCR data prevents the "hallucination" of text corrections.
• Comparison: The method outperforms both standard fine-tuning and uniform distillation, achieving the best trade-off between linguistic recovery and visual capability retention.

5. Significance and Conclusion

LINGUDISTILL provides a practical and efficient solution to the problem of modality-specific degradation in multimodal models.

• Efficiency: It avoids the memory and latency overhead of adding adapter modules, making it suitable for deployment in resource-constrained environments.
• Insight: The work highlights that multimodal adaptation perturbs linguistic priors in a way that simple fine-tuning cannot fix, but that these priors can be "distilled back" if the teacher is given access to the multimodal context via KV sharing.
• Generalizability: The approach suggests that targeted cross-modal distillation is a viable path for preserving backbone capabilities in future multimodal systems without architectural bloat.

In summary, LINGUDISTILL proves that with the right distillation strategy and architectural trick (KV sharing), one can have a VLM that is both visually grounded and linguistically robust, without the cost of extra parameters.