Technical Summary: Improving MLLM Training Efficiency via Stage-Aware Sparsity

Problem Statement

Multimodal Large Language Models (MLLMs) have demonstrated significant capabilities but suffer from inefficient training processes. This inefficiency stems from two primary sources of redundancy:

1. Data Redundancy: Multimodal inputs introduce hundreds of visual tokens per sample, many of which contribute marginally to final predictions.
2. Parameter Redundancy: Deep transformer architectures contain dozens of layers, yet not all layers are equally critical throughout the entire training process.

Existing efficiency methods typically address these issues independently (e.g., token compression or layer skipping) and treat redundancy as a static property. However, the paper argues that redundancy patterns are stage-dependent. During modality alignment, the language model (LLM) is often frozen, and the bottleneck is processing massive visual token sequences. Conversely, during instruction tuning, the LLM parameters are actively updated, and the bottleneck shifts to the computational overhead of deep transformer layers. Treating redundancy as static leads to suboptimal efficiency-performance trade-offs.

Methodology: Sparse Training Scheme (STS)

The authors propose the Sparse Training Scheme (STS), a unified framework that adapts sparsity strategies to the specific computational bottlenecks of different training stages. STS consists of two complementary components:

1. Visual Token Compressor (VTC) – Stage I: Modality Alignment

Applied during the modality alignment stage where the LLM is frozen and only the projector is updated.

• Function: Reduces the length of visual tokens ($T_v$) before they enter the LLM to alleviate data redundancy.
• Mechanism: A selector function $C$ maps $T_v \in \mathbb{R}^{b \times n \times d}$ to a compressed sequence $T_v' \in \mathbb{R}^{b \times n' \times d}$ ($n' < n$).
• Selection Strategies:
  • Uniform Sampling: Fixed stride selection to preserve global semantic structures.
  • Random Sampling: Introduces stochasticity to improve robustness.
  • Instruction-Guided: Uses cosine similarity between visual and text tokens to discard irrelevant visual information.
• Implementation: The paper adopts Uniform Sampling as the default strategy, finding it provides the most balanced performance across tasks compared to the instability of random sampling or the bias of instruction-guided selection.

2. Layer Dynamic Skipper (LDS) – Stage II: Instruction Tuning

Applied during instruction tuning where LLM parameters are updated.

• Function: Dynamically skips unnecessary decoder layers to reduce parameter updates and computational overhead.
• Mechanism: For layer $l$ at training step $e$, the hidden state is updated as $H_l = (1 - z_l) \cdot F_l(H_{l-1}) + z_l \cdot H_{l-1}$, where $z_l$ is a Bernoulli variable controlled by a skip probability $p_l(e)$.
• Probability Formulation: The skip probability combines two factors:
  • Step-based Decay: Based on the "coarse-to-fine" convergence trend. Early in training, gradients are coarse, allowing higher skip rates. The probability decays quadratically as training progresses: $\alpha (\frac{E-e}{E})^2$.
  • Depth-based Adjustment: Acknowledging that shallow layers extract features more effectively than deep layers, the probability is adjusted by layer depth $l$: $1 + \epsilon \cdot \frac{2l - (L-1)}{L-1}$.
• Default Configuration: The authors find $\alpha=0.5$ and $\epsilon=0.5$ provide the best balance between efficiency and task performance.

Key Contributions

1. Analysis of Stage-Dependent Redundancy: The work reveals that redundancy in MLLM training is not static; data redundancy dominates modality alignment, while parameter redundancy dominates instruction tuning.
2. Unified Framework (STS): Proposes a stage-aware sparse training framework that explicitly models evolving redundancy patterns, applying token-level sparsity during alignment and layer-level sparsity during tuning.
3. Empirical Validation: Demonstrates through extensive experiments that STS significantly reduces computational costs while maintaining high accuracy across diverse MLLM architectures.

Experimental Results

The authors evaluated STS on five MLLMs (LLaVA, Mipha, Phi-2, MoE-LLaVA variants) across benchmarks including GQA, VQAv2, SQA, POPE, and MME.

• Efficiency: STS achieved an average reduction of over 15% in FLOPs. Specifically, the theoretical FLOPs ratio relative to the baseline was reduced to approximately 82.2%.
• Performance: The method maintained 96.9% to 99.7% of baseline accuracy on average.
  • Robustness metrics (POPE) and comprehensive evaluations (MME) showed minimal degradation.
  • Perception-oriented tasks (GQA, VQAv2) showed slightly more noticeable drops (around 2–3 points), though in some cases (e.g., SQA on LLaVA), STS outperformed the baseline, suggesting potential implicit regularization effects.
• Ablation Studies:
  • VTC: Uniform sampling proved more robust than random or instruction-guided sampling.
  • LDS: Extreme values for hyperparameters $\alpha$ (skip rate) and $\epsilon$ (depth variance) biased the model toward specific capabilities. The balanced configuration ($\alpha=0.5, \epsilon=0.5$) yielded the best trade-off.
  • Joint Framework: Using VTC and LDS together provided a superior efficiency-performance trade-off compared to using either component in isolation.

Significance and Claims

The paper claims that STS offers a practical and scalable solution for efficient MLLM optimization. By shifting the focus from static redundancy reduction to a stage-aware approach, the framework addresses the specific bottlenecks of different training phases without requiring complex architectural changes. The authors emphasize that STS is broadly applicable to diverse MLLM architectures and achieves faster training with minimal performance loss, making it a viable strategy for reducing the computational barriers in training large-scale multimodal models.