Technical Summary: HEED - Density-Weighted Residual Alignment for Hybrid Vision-Language Model Distillation

1. Problem Statement

The distillation of large Vision-Language Models (VLMs) into efficient hybrid architectures (e.g., mixing Mamba-2 linear layers with attention layers) is a standard approach for reducing inference costs. However, existing aggregate benchmarks mask a critical failure mode: while hybrid students often preserve broad reasoning capabilities, they suffer significant performance drops on tasks requiring fine-grained visual perception, such as Optical Character Recognition (OCR), document understanding, and chart analysis.

The authors identify that standard knowledge distillation (KD) treats all visual tokens uniformly. In high-resolution images, most patches consist of smooth backgrounds (sky, walls) with low information density, while a small fraction contains critical details (text, edges, object boundaries). Standard KD allocates excessive loss terms to low-information background patches, failing to protect the sparse, high-density patches essential for answering specific questions. This results in a "fine-grained perception collapse" where the student can describe a scene generally but fails to read specific digits or labels within it.

2. Methodology: HEED

The proposed method, HEED (Density-Weighted Residual Alignment), addresses this imbalance by reweighting the distillation loss based on the local information density of visual patches.

2.1 Diagnostic Insight

Through token-level diagnostics, the authors localize the failure to specific positions:

• Residual Drift: The top 10% of patches with the highest local feature density (defined by ViT feature dissimilarity from neighbors) exhibit 3.6× larger residual drift from the teacher compared to the bottom 10%.
• Answer Sensitivity: Masking these high-density patches causes a 3.5× larger drop in the teacher's answer score compared to low-density patches.
• Predictive Power: Local visual density is the strongest predictor of residual drift, outperforming token type (text vs. visual), layer depth, and teacher attention concentration.

2.2 The HEED Algorithm

HEED implements a training-free, density-weighted residual alignment strategy without modifying the model architecture or adding parameters:

1. Density Estimation: For each visual patch $p$, a density score $\rho(p)$ is computed using the self-dissimilarity of frozen ViT features within a $3\times3$ neighborhood:
   $$\rho(p) = 1 - \frac{1}{|N(p)|} \sum_{q \in N(p)} \frac{v_p^\top v_q}{\|v_p\| \|v_q\|}$$
   High $\rho(p)$ indicates a locally distinctive patch (e.g., text, edges); low $\rho(p)$ indicates a smooth background.
2. Weighting: These scores are normalized and converted into loss weights $w(p)$ using an exponential mapping with a temperature parameter $\tau$. Text tokens receive a constant boost based on the image's average visual density.
3. Residual Alignment: Instead of aligning per-layer block outputs (standard hidden-state alignment) or using uniform weights, HEED aligns the residual streams of the teacher and student. The alignment loss is weighted by the density scores:
   $$L_{HEED} = \frac{1}{|S^*|} \sum_{\ell \in S^*} \sum_{p=1}^{T} w(p) \| r_{\ell,p}^{\tilde{\theta}} - r_{\ell,p}^{\theta} \|_2^2$$
   This ensures that the hybrid student prioritizes preserving information at high-density positions during the distillation process.

2.3 Training Pipeline

The method follows a three-stage distillation schedule:

• Stage 1 (Warm-up): Aligns residual streams with frozen attention weights.
• Stage 2 (Full Block): Trains full Mamba-2 blocks with density-weighted residual alignment.
• Stage 3 (End-to-End): Standard logit-based KD (KL divergence + Cross-Entropy).
  HEED differs from baseline methods only in the alignment loss used in Stages 1 and 2.

3. Key Contributions

1. Diagnosis: The paper demonstrates that hybrid VLM distillation selectively fails on fine-grained perception tasks (OCR, documents) while preserving reasoning, attributing this to the uniform treatment of tokens in standard KD.
2. Mechanism: It establishes that high-density visual positions (locally distinctive features) are the primary source of drift and are most critical for the teacher's answer. Random reweighting of arbitrary patches does not yield the same benefit; the weight must specifically target high-density positions.
3. Method (HEED): A lightweight, parameter-free intervention that uses patch self-dissimilarity as a proxy for importance. It improves performance without altering the inference architecture or adding inference-time costs.

4. Experimental Results

Experiments were conducted by distilling Qwen3-VL-8B-Instruct into a 3:1 Mamba-2/Attention hybrid.

• Performance Gains:
  • Compared to standard end-to-end KD, HEED improves OCRBench v2 by 8.7 points.
  • It achieves a 5.13-point average gain across 10 benchmarks.
  • In a controlled comparison (changing only the alignment weight from uniform to density-weighted), HEED recovers 4.7 points on OCRBench v2 and 2.24 points on the 10-benchmark average.
• Selectivity: The gains are concentrated in fine-grained perception benchmarks (OCR, DocVQA, ChartQA), while reasoning benchmarks (MMMU-Pro, MMStar) remain stable or slightly improved.
• Post-Training: After standard SFT and DPO, the HEED student reaches teacher-level performance on the 10-benchmark average (78.79 vs. 78.41) while maintaining the hybrid architecture's efficiency.
• Efficiency: The student retains the hybrid model's efficiency profile: 4.12× throughput and 68% memory savings at 128k context length compared to the teacher, with no additional inference-time cost.
• Robustness: The results hold across different random seeds, different teacher models (InternVL-3.5, MiniCPM-V 4.5, GLM-4.6V), and different hybrid architectures (Gated DeltaNet).

5. Significance and Claims

The paper claims that HEED addresses a fundamental limitation in compressing heterogeneous multimodal inputs into fixed-capacity hybrid sequence models. The core principle is that distillation losses should not treat all tokens uniformly; instead, they must reflect the varying importance of tokens based on their local information density.

The authors position HEED as complementary to inference-time token reduction methods (e.g., VisionZip, FastV). While those methods reduce the number of tokens during inference, HEED optimizes the training signal to ensure the model learns to preserve the most critical tokens during the conversion from a full-attention teacher to a hybrid student. The method is presented as a practical, low-cost intervention that recovers lost fine-grained capabilities without compromising the efficiency gains of hybrid architectures.

Limitations Acknowledged:

• Density is a proxy for importance, not a universal semantic saliency measure. It favors text and edges but may be less effective for globally defined or adversarially patterned tasks.
• The method relies on frozen ViT features; it does not claim to be a causal proof of drift but rather a highly effective empirical intervention.
• The results are specific to the hybrid VLM conversion setting tested; the authors do not claim Pareto dominance over full-attention models generally.