Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM

Technical Summary: Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM

Problem Statement
Multimodal Large Language Models (MLLMs) currently rely heavily on Transformer architectures, which suffer from quadratic computational and memory complexity relative to input sequence length. This bottleneck limits the scalability of MLLMs and imposes unsustainable computational loads. While the Selective Scan Structured State-Space Model (Mamba) offers a promising alternative with linear complexity and efficient inference, its application to multimodal modeling remains under-explored. Specifically, existing attempts to integrate Mamba into MLLMs face two primary challenges: (1) the necessity of manually designing heuristic 2D scan orders (e.g., bidirectional or cross-directional) to convert image features into 1D sequences, and (2) the lack of flexibility in downsampling visual token lengths, often resulting in excessive sequence lengths that negate Mamba's efficiency benefits.

Methodology
The authors propose Querying Mamba, a novel architecture that introduces a query-based cross-modal projector to bridge pre-trained vision encoders and Mamba-based LLMs. The methodology is structured around three core components:

1. Q-Mamba Projector Architecture:

   • The projector utilizes a stack of Q-Mamba blocks, each comprising a Mamba layer, a cross-attention mechanism, and a feedforward network.
   • Learnable Queries: Instead of processing raw image tokens directly, the model employs a set of learnable query embeddings. These queries interact with visual features from a frozen pre-trained vision encoder (SigLIP) via cross-attention.
   • Causal Prior: A unidirectional Mamba layer is applied to the queries to introduce causal dependencies, ensuring the projected sequence is compatible with the sequential processing of the LLM.
   • Local Attention Mask: The cross-attention mechanism employs a local attention mask, where each query attends exclusively to a specific subset of keys, facilitating focused extraction from distinct visual components.

2. Visual Token Compression and Alignment:

   • The projector dynamically compresses 2D visual information into a 1D causal token sequence.
   • By adjusting the number of learnable queries, the model can effectively downsample the visual feature sequence length, reducing the computational burden on the LLM backbone without requiring manual scan order definitions.

3. Multimodal Mamba LLM Training:

   • The system integrates a pre-trained Mamba LLM (2.8B parameters) with the vision encoder via the Q-Mamba projector.
   • Training follows a two-stage paradigm:
     • Stage 1 (Alignment): The projector is aligned with the frozen LLM using a filtered visual instruction-following dataset (CC3M).
     • Stage 2 (End-to-End Fine-tuning): Both the projector and the LLM are fine-tuned on a combined dataset (LLaVA v1.5, LVIS-Instruct-4V, LRV-Instruct) to enhance vision-language integration and mitigate hallucinations.

Key Contributions

• Querying Mamba: The introduction of a Mamba-based multimodal connector that utilizes cross-attention and learnable queries to adaptively downsample visual token lengths, eliminating the need for heuristic 2D scan orders.
• Architectural Exploration: A systematic investigation into the components of the multimodal connector, including the use of bidirectional Mamba for preprocessing, feedforward networks for channel mixing, and optimal query sequence lengths.
• Comprehensive Evaluation: Extensive benchmarking of the proposed MLLM across six diverse vision-language understanding tasks, demonstrating improvements in both performance and throughput compared to existing Mamba-based MLLMs.

Experimental Results
The proposed model was evaluated on six benchmarks: VQAv2, GQA, VizWiz, VQAT, POPE, and MMBench.

• Performance: The model with 729 learnable queries achieved the highest overall performance, outperforming state-of-the-art Mamba-based models (Cobra and VL-Mamba) across most metrics. For instance, it achieved a score of 75.62 on VQAv2 and 58.33 on GQA.
• Query Length Analysis: Increasing the query count from 128 to 256 yielded substantial gains. However, further increases to 512 and 729 showed diminishing returns. Notably, metrics for VizWiz (unanswerable questions) and POPE (hallucination) exhibited an inverse relationship with query size, suggesting that excessive queries may introduce extraneous data that complicates decision-making.
• Throughput: The dynamic downsampling capability significantly reduced computational load. The proposed model achieved faster training (5.52 sec/iter) and inference (0.095 sec/iter) times compared to Cobra (7.62/0.129) and VL-Mamba (7.26/0.152).
• Ablation Studies:
  • Attention Mechanism: Local attention significantly outperformed global attention.
  • Pre-training: Utilizing weights from a trained VL-Mamba for the bidirectional connector improved performance over training from scratch.
  • Scan Order: Interestingly, while the model was trained with bidirectional scanning, inference using only a forward scan did not degrade performance and, in some cases, improved it.

Significance and Claims
The paper claims that the proposed query-based cross-modal projector effectively bolsters Mamba's efficiency for vision-language modeling. By compressing visual tokens based on input context through cross-attention, the architecture removes the dependency on manually designed 2D scan orders, a limitation in previous Mamba-based vision encoders. The authors assert that this approach enhances both the performance and throughput of Mamba-based MLLMs, offering a scalable solution for multimodal tasks.

Limitations
The authors acknowledge several limitations:

• Dataset Constraints: The training and fine-tuning datasets (LLaVA-LLVIS, LLaVA-1.5) are smaller than those typically used for large language models, potentially restricting generalization.
• Training Duration: The model was trained for only one epoch per stage, whereas comparable models often use two, which may impact robustness.
• Mamba "Forgetting": The sequential nature of Mamba's hidden states may lead to forgetting issues with long inputs, similar to RNNs.
• Future Work: The authors suggest that more extensive pre-training of Q-Mamba (including contrastive learning) and initialization with compact Mamba LLM weights could further improve performance. They also note the potential for bias and hallucination inherent in large language models.