Technical Summary: SAB-LVLM

Problem Statement

Large Vision-Language Models (LVLMs) have achieved remarkable progress in multimodal understanding but suffer from enormous parameter scales and cross-modal computational costs, leading to significant memory and latency overheads that hinder deployment on resource-constrained devices. While binarization offers a solution by reducing weights to approximately 1-bit, existing Post-Training Quantization (PTQ) methods designed for Large Language Models (LLMs) fail to address the unique challenges of LVLMs. Specifically, current methods often minimize global weight quantization error without considering the heterogeneous importance of weights across different layers and modalities. This oversight leads to the retention of task-irrelevant encoding parameters while failing to adequately optimize modality-critical weights, resulting in severe performance degradation when applied to LVLMs.

Methodology

The authors propose SAB-LVLM (Significance-Aware Binarization for Large Vision-Language Models), a framework designed to preserve task-relevant information during binarization by distinguishing between weights activated by single modalities and those activated across modalities. The methodology consists of three core components:

1. Significance-Aware Binarization Map Construction:

   • Hessian-Based Sensitivity: The method first constructs Hessian matrices for textual and visual inputs separately using calibration data. These matrices are used to compute a sensitivity matrix $S$ for each weight, quantifying how sensitive the weight is to perturbations in each modality.
   • Spatial Significance Map ($\Gamma$): To distinguish between unimodal and multimodal activations, the authors compute a spatial significance map $\Gamma$. This map measures the sparsity of the sensitivity vector for each weight. Weights activated primarily by a single modality (unimodal) receive high significance values ($\Gamma \to 1$), while weights activated by inputs from different modalities (multimodal) receive low values ($\Gamma \to 0$).
   • Modality-Guided Significance Integration (MGSI): Recognizing that binarization should not solely favor unimodal weights, the authors introduce a modality integration score $r$. This score dynamically balances the importance of unimodal versus multimodal weights. The final significance-aware binarization map $\Lambda$ is computed as a weighted combination of the spatial significance map and its complement, controlled by $r$.

2. Error Reweighting Objective:
   Unlike standard binarization objectives that minimize the Frobenius norm of the difference between full-precision and binary weights, SAB-LVLM incorporates the significance map $\Lambda$ as an error reweighting term. The objective function becomes $\|\sqrt{\Lambda} \odot (W - \hat{W})\|_F^2$. Theoretically, this ensures that weights with higher significance (larger $\Lambda$) incur smaller quantization errors, thereby preserving the model's representational capacity for critical tasks.

3. Alternating Significance-Weighted Update:
   To optimize the binary weights under this new objective, the authors employ an alternating refinement scheme. This involves iteratively updating row-wise and column-wise scaling vectors ($\alpha_r, \alpha_c$) and binary bases ($B$) while incorporating the significance map $\Lambda$ into the gradient updates. This process allows for a precise fitting of the binary weights to the full-precision weights, prioritizing the preservation of significant parameters.

Key Contributions

• Early Exploration of LVLM Binarization: The paper presents one of the first attempts to achieve weight binarization specifically for Large Vision-Language Models, addressing the gap in existing PTQ research which focuses primarily on LLMs.
• Spatial Significance Map: A novel mechanism to distinguish between weights activated under a single modality versus those activated across modalities, enabling the model to handle the heterogeneous nature of LVLM parameters.
• Modality-Guided Integration: A strategy to integrate unimodal and multimodal significance scores dynamically, ensuring that the binarization process respects the complementary interactions between visual and textual modalities.
• Alternating Significance-Weighted Update: An optimization scheme that incorporates the significance map directly into the binarization fitting process, theoretically guaranteeing that higher-significance weights are preserved with lower quantization error.

Experimental Results

Extensive experiments were conducted on representative LVLM benchmarks, including MMStar, DocVQA, TextVQA, Video-MME, and VSI-Bench, using models from the Qwen2.5-VL (7B, 32B, 72B) and InternVL3.5 (8B, 14B, 38B) families.

• Performance Superiority: Under an approximate 1-bit compression constraint, SAB-LVLM consistently outperformed existing binary PTQ baselines (including PB-LLM, BiLLM, and ARB-LLM) across all tested models and benchmarks. For instance, on the Qwen2.5-VL-7B-Instruct model, SAB-LVLM achieved an MMStar score of 45.79, significantly surpassing ARB-LLM (38.10) and BiLLM (29.97).
• Task Preservation: The method demonstrated superior capability in preserving coarse perception, fine-grained perception, instance reasoning, and logical reasoning compared to baselines. In qualitative analysis, SAB-LVLM successfully answered questions regarding object counting and reasoning where baselines failed.
• Ablation Studies:
  • The spatial significance map was found to be sensitive to the threshold $\tau$, with $\tau=0.0003$ yielding optimal results.
  • The modality-guided significance integration (MGSI) outperformed fixed integration scores and single-modality sensitivity approaches, confirming the necessity of jointly modeling both modalities.
  • The full spatial significance map (w/ SAB) achieved higher scores than row-wise or column-wise reweighting variants, validating the importance of element-wise significance.

Significance and Claims

The paper claims that SAB-LVLM represents a significant step forward in compressing LVLMs for real-world deployment. By explicitly addressing the varying importance of weights across layers and modalities, the method mitigates the severe performance degradation typically associated with binarizing multimodal models. The authors assert that their approach strikes a favorable trade-off between extreme compression efficiency (approx. 1-bit) and task performance, making LVLMs more viable for resource-constrained environments without the need for costly retraining. The work is positioned as an early exploration, with future work planned to evaluate the framework on broader architectures and more diverse multimodal tasks.