From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion

Technical Summary: From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion

Problem Statement
Current Vision-Language Models (VLMs) suffer from a fundamental architectural bottleneck: they rely on a static, asymmetric "one-to-one" connection that links only the final output layer of the Vision Transformer (ViT) encoder to the input of the Large Language Model (LLM). This design discards the rich hierarchical information captured in intermediate ViT layers, such as low-level textures and edges, which are crucial for fine-grained perception. Consequently, LLMs are denied access to the full spectrum of visual evidence, limiting their ability to integrate local details with global semantics for nuanced reasoning. The paper argues that this static connection creates a "functional mismatch," preventing models from performing sophisticated tasks like identifying specific object parts (e.g., wheels on a shoe) or grounding abstract concepts in fine-grained visual data.

Methodology: Cross-Layer Injection (CLI)
To resolve this, the authors propose Cross-Layer Injection (CLI), a framework that replaces the static bridge with a dynamic "many-to-many" architecture. This allows the LLM to actively query and integrate features from the entire visual hierarchy based on its real-time decoding context. CLI comprises two synergistic, parameter-efficient components:

1. Adaptive Multi-Projection (AMP): To address the distributional variance between features from different ViT layers (ranging from low-level textures to high-level semantics), the authors introduce a projection module that harmonizes these features into a shared semantic space. Instead of training a separate projector for each layer (which is computationally prohibitive), AMP utilizes Low-Rank Adaptation (LoRA). It augments the pre-trained projector with layer-specific LoRA adapters, allowing efficient alignment of multi-level visual tokens with the LLM's embedding space.

2. Adaptive Gating Fusion (AGF): This is the core mechanism enabling dynamic selection. At multiple injection points within the LLM decoder, an AGF module acts as a context-sensitive controller. It employs a query-based attention mechanism where learnable query vectors distill information from both the current LLM hidden state and the aligned visual features. A gate controller computes a dynamic weight (0 to 1) to determine the relevance of the new visual information. This allows the LLM to selectively "zoom in" on fine-grained details from shallow ViT layers or "zoom out" to abstract concepts from deep layers, depending on the specific reasoning step required.

Key Contributions

• A Novel Framework: CLI systematically addresses the underutilization of hierarchical visual features by establishing a dynamic many-to-many bridge between vision and language hierarchies.
• Synergistic Modules: The introduction of AMP and AGF, which collectively enable the alignment and selective, context-sensitive fusion of multi-level visual features without requiring massive parameter increases.
• Empirical Validation: Extensive experiments on two distinct VLM architectures (LLaVA-OneVision and LLaVA-1.5) across 28 diverse benchmarks demonstrate consistent and significant performance improvements over baselines and other deep fusion strategies.

Results
The authors integrated CLI into LLaVA-OneVision (0.5B and 7B parameters) and LLaVA-1.5 (7B parameters). The results across 28 benchmarks covering document analysis, chart reasoning, and general perception show:

• Significant Gains: CLI outperforms the baseline projector and other fusion methods (such as DeepStack and Shallow-Layer Injection). For instance, on LLaVA-OV-7B, CLI achieved improvements of 6.5 points on LLaVA-in-the-Wild, 3.3 points on MME, and 4.7 points on OCR-Bench.
• Superiority over Alternatives: Unlike DeepStack, which often degrades performance due to non-selective, brute-force injection, and Shallow-Layer Injection, which fails to aid deep reasoning layers, CLI's dynamic gating ensures that only relevant visual information is integrated.
• Fine-Grained Performance: The method shows particular strength in tasks requiring high-fidelity detail, such as Optical Character Recognition (OCR) and visual grounding, where access to early ViT layers is critical.
• Efficiency: The framework introduces minimal computational overhead, with inference memory increasing by only 1.3%, while delivering substantial performance gains.

Significance and Claims
The paper claims that CLI establishes a new scalable paradigm for deep multimodal understanding. By granting LLMs on-demand access to the full visual hierarchy, the framework unlocks a deeper level of reasoning that static architectures cannot achieve. The authors emphasize that the performance gains stem from the architectural effectiveness of the dynamic many-to-many fusion rather than merely an increase in parameter count. The work suggests that for VLMs to approach human-like cognition, they must move beyond static, single-layer connections to dynamic, context-aware interactions that allow the language model to act as an "active observer" of the visual world.