Mitigating Visual Degradation in MLLMs via Spatial-Spectral Visual Anchor Learning

Technical Summary: Mitigating Visual Degradation in MLLMs via Spatial-Spectral Visual Anchor Learning

Problem Statement
Despite significant advancements in Multimodal Large Language Models (MLLMs) through visual instruction tuning, these models exhibit persistent deficiencies in visual perception. The authors identify a critical, under-explored phenomenon: visual representation deviation. Following visual instruction tuning, internal visual representations within the Large Language Model (LLM) layers progressively deviate from their original semantic states during inference. This leads to severe information degradation in deeper layers, causing the model to lose fine-grained spatial details and semantic fidelity.

The paper challenges the prevailing intuition that directly aligning intermediate LLM representations with external Vision Foundation Models (VFMs) solves this issue. While such direct alignment (e.g., VIRAL) enhances the quality of visual features, the authors demonstrate that it fails to mitigate the deviation itself; representations still diverge at a similar rate across layers. The core problem is not merely what the representations encode, but how stably they are maintained as they propagate through the network.

Methodology: Spatial-Spectral Visual Anchor Learning (SSVAL)
To address representation deviation without incurring inference overhead, the authors propose SSVAL, a framework centered on Visual Anchor Prompt Injection (VAPI) and auxiliary Spatial-Spectral Representation Alignment (SSRA).

1. Visual Anchor Prompt Injection (VAPI):

   • Mechanism: The core innovation is a set of learnable visual prompts that act as "knowledge vessels." During training, these prompts absorb rich visual knowledge from an external VFM (e.g., DINOv2, CLIP) via a gated multi-head cross-attention mechanism.
   • Function: The prompts are initialized randomly and optimized to capture VFM features. During inference, the external VFM is discarded. The learned prompts function as stable visual anchors, providing consistent reference points for each LLM layer to arrest the drift of visual representations.
   • Architecture: Vision features serve as queries, while the combined prompts (original prompts + projected VFM features) serve as keys and values in a cross-attention module, followed by a gated residual connection.

2. Spatial-Spectral Representation Alignment (SSRA):

   • To provide complementary vision-specific supervision at intermediate LLM layers (which standard text-based training lacks), SSVAL introduces two auxiliary loss functions:
     • Spatial Representation Alignment (SpaRA): Aligns student (LLM) and teacher (VFM) features from a multi-scale spatial perspective (pooling factors 1x1, 2x2, 4x4). It maximizes cosine similarity for feature-level alignment and constrains pairwise spatial relationships for relation-level alignment.
     • Spectral Representation Alignment (SpeRA): Operates in the frequency domain. It decomposes features into Low-Frequency (LF) and High-Frequency (HF) components using 2D Fast Fourier Transforms (FFT). LF components capture global structure, while HF components capture edges and local details. The loss applies Mean Squared Error (MSE) to LF and logarithmic compression to HF to prevent noise dominance.

3. Training Objective:
   The total loss combines the standard next-token prediction loss with the weighted SpaRA and SpeRA alignment losses. This joint optimization ensures the prompts absorb VFM knowledge while the alignment losses reinforce visual integrity at intermediate layers.

Key Contributions

• Identification of Deviation: The paper quantifies visual representation deviation in MLLMs, showing that cosine similarity between input and intermediate representations drops sharply with layer depth. It reveals that direct VFM alignment fails to stop this drift.
• Visual Anchor Prompt Injection (VAPI): A novel mechanism using prompts as persistent anchors that absorb VFM knowledge during training and stabilize representations during inference, effectively mitigating deviation without external models at test time.
• Spatial-Spectral Supervision: The introduction of SpaRA and SpeRA provides complementary multi-scale spatial and frequency-domain supervision, addressing the lack of explicit visual guidance in intermediate layers.
• Efficiency: The method introduces negligible inference overhead (only +1.55% parameters and +0.04% FLOPs) as the VFM and alignment losses are training-only components.

Experimental Results
Extensive experiments were conducted on six multimodal benchmarks (CV-Bench2D, MMVP, MMStar, MME, POPE, MM-Bench) using different vision encoders (CLIP, SigLIP) and LLM scales (Qwen2.5-3B, Qwen2.5-7B).

• Performance: SSVAL significantly outperforms both the standard Baseline (Visual Instruction Tuning) and the direct alignment method (VIRAL). Notable improvements include +6.47% on CV-Bench2D and +7.86% on MMVP with the Qwen2.5-7B backbone.
• Robustness: Unlike VIRAL, which showed performance drops on certain benchmarks (e.g., -5.34% on MMVP with SigLIP), SSVAL demonstrated consistent gains, suggesting its anchor mechanism is more robust to varying initial visual representation qualities.
• Ablation Studies:
  • VAPI was identified as the primary driver of performance gains.
  • SpaRA and SpeRA provided complementary benefits, with multi-scale spatial alignment and the combination of LF/HF frequency components yielding the best results.
  • Prompt Quantity: Four prompts were found to be optimal for the tested configurations.
  • VFM Source: Using DINOv2-B as the knowledge source yielded the best overall balance, though DINOv2-L performed better on specific reasoning benchmarks.

Qualitative Analysis
Qualitative results show that SSVAL produces embeddings with clearer clustering boundaries and stronger structural organization compared to baselines. Attention maps indicate that SSVAL maintains focused attention on relevant objects, whereas baselines exhibit scattered attention patterns. The learned prompts were observed to develop specialized roles: one prompt suppressing background noise and others inheriting foreground objectness from the VFM, effectively bridging the gap between different VFM capabilities.

Significance
The paper claims that SSVAL offers a fundamental shift in addressing visual degradation: rather than simply injecting stronger features, it establishes a persistent anchor mechanism that stabilizes representations throughout the network. By separating knowledge absorption (training) from inference efficiency, SSVAL achieves substantial performance improvements in fine-grained perception and spatial reasoning with minimal computational cost, validating the hypothesis that visual anchors are essential for maintaining representation integrity in deep MLLMs.