🤖 machine learning

Fast-Slow Efficient Training for Multimodal Large Language Models via Visual Token Pruning

This paper proposes DualSpeed, a fast-slow training framework that combines visual token pruning for efficiency with a full-sequence auxiliary mode and self-distillation to resolve training-inference mismatches, thereby accelerating MLLM training by up to 4.0 $\times$ without compromising performance.

Original authors: Dingkun Zhang, Shuhan Qi, Yulin Wu, Xinyu Xiao, Xuan Wang, Long Chen

Published 2026-02-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dingkun Zhang, Shuhan Qi, Yulin Wu, Xinyu Xiao, Xuan Wang, Long Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart student (a Multimodal Large Language Model) to understand the world by showing them thousands of pictures.

The Problem: The "Too Much Information" Bottleneck
Right now, these models are like students who get overwhelmed. When you show them a picture, the computer breaks it down into hundreds or even thousands of tiny pieces called "visual tokens." It's as if you handed the student a 1,000-page book for every single image, even though most of the pages are just blank white space or repetitive patterns.

Trying to read all those pages takes forever. This makes training these models incredibly slow, expensive, and energy-hungry.

The Old Idea: The "Cut-and-Paste" Mistake
Researchers realized that many of those pages are useless. They tried a technique called Visual Token Pruning, which is like using a highlighter to cross out the boring, redundant pages before showing the book to the student. This works great for testing the student later (inference), making them faster.

But when they tried using this method during training, it backfired. It created a "mismatch."

The Analogy: Imagine you only ever trained the student on a 10-page summary of a book. Then, on the final exam, you handed them the full 1,000-page book. The student would panic and fail because they never learned how to handle the full version. They got too used to the short version.

The Solution: DualSpeed (The "Fast-Slow" Training Camp)
The authors of this paper, Dingkun Zhang and team, created a new training framework called DualSpeed. Think of it as a two-track training camp that switches back and forth randomly to solve the mismatch problem.

The Fast Lane (The Speedy Practice):
- Most of the time (about 90% of the sessions), the model trains in "Fast Mode."
- Here, they use the "pruning" technique to cut out the useless visual tokens. The model learns quickly from the short, efficient summaries.
- To help the model remember which version it's looking at, they add a tiny, invisible "name tag" (called a Mode Isolator) to the start of the short summary. This tells the model, "Hey, this is the condensed version; focus on the key details."
The Slow Lane (The Full-Book Review):
- Occasionally (about 10% of the time), the model switches to "Slow Mode."
- Here, they show the model the full, unpruned picture (all 1,000 pages).
- To make sure the model doesn't get lazy during these rare sessions, they use a trick called Self-Distillation. The "Fast Mode" (which has already learned a lot) acts as a teacher, whispering the answers to the "Slow Mode" student. This ensures the student learns the full version effectively, even though they see it less often.

The Result: Best of Both Worlds
By mixing these two modes, the model gets the speed of the Fast Lane but retains the ability to handle the full, complex world of the Slow Lane.

Speed: They trained models 2.1 times to 4.0 times faster than before.
Performance: Despite the speed, the models didn't get dumber. They kept over 99% of their original performance.
Flexibility: The final model is smart enough to handle both the short summaries (if you want speed later) and the full images (if you want maximum accuracy).

In Summary
The paper claims that by using a "Fast-Slow" switching system, they can teach these massive AI models much faster without making them forget how to read the full books. They found that roughly 90% of the visual tokens are actually redundant during training, and by cutting them out smartly (while occasionally practicing with the full version), they can save massive amounts of time and money without losing intelligence.

Technical Summary: DualSpeed for Efficient MLLM Training

1. Problem Statement

Multimodal Large Language Models (MLLMs) face a critical bottleneck in training efficiency due to their massive model sizes and the sheer volume of visual tokens generated from high-resolution inputs. While existing efficient training methods focus on reducing model parameters or trainable parameters, they often overlook the potential of reducing the number of training tokens.

Visual Token Pruning (VTP) has proven effective for improving inference efficiency by discarding redundant visual tokens (e.g., background or repetitive textures). However, directly applying VTP during the training phase introduces a severe training-inference mismatch. Models trained on pruned sequences struggle to perform effectively when inferring on full, non-pruned visual sequences. This gap arises from discrepancies in sequence length, information density, and spatial structure between the training and inference data distributions. Consequently, simply pruning tokens during training leads to significant performance degradation in standard inference scenarios.

2. Methodology: DualSpeed

To address the training-inference mismatch while leveraging the efficiency of VTP, the authors propose DualSpeed, a fast-slow framework for MLLM training. The framework operates by randomly switching between two distinct modes during the training process:

A. Fast-Mode (Primary Efficiency Mode)

Objective: Maximize training speed by reducing visual tokens.
Mechanism: This mode incorporates existing VTP methods as plugins to prune the visual token sequence.
Mode Isolator: To ensure the model can distinguish between pruned and full inputs, a learnable Mode Isolator (a soft prompt of length $l$ ) is concatenated as a prefix to the pruned visual sequence. This explicitly prompts the LLM to activate a specific perception pattern suitable for sparse inputs. When processing full sequences, this isolator is disabled.
Training Objective: Standard Cross-Entropy loss on the pruned sequence (with the isolator).

B. Slow-Mode (Auxiliary Consistency Mode)

Objective: Maintain training-inference consistency by learning from full visual sequences.
Mechanism: The model is trained on non-pruned, full visual sequences.
Self-Distillation: Since the slow-mode is activated less frequently (controlled by probability $r$ ), its learning is boosted via self-distillation. The sufficiently trained Fast-Mode acts as a "teacher," and the Slow-Mode acts as a "student." They share the same model parameters but differ in input sequences. The student minimizes the Kullback–Leibler (KL) divergence between its output logits and the teacher's logits (computed on the pruned sequence).
Training Objective: A combination of Cross-Entropy loss on the full sequence and the distillation loss.

C. Overall Training Strategy

The framework switches between modes per batch based on a probability $r$ . During phases where the LLM backbone is frozen (e.g., pretraining), only the Fast-Mode is activated to further enhance efficiency. During inference, the model operates exclusively on full, non-pruned visual sequences without the Mode Isolator, ensuring uncompromised performance.

3. Key Contributions

Identification of a Key Challenge: The paper identifies and analyzes the training-inference mismatch as a primary barrier to applying Visual Token Pruning during the training of MLLMs.
DualSpeed Framework: The proposal of a collaborative fast-slow training framework that integrates VTP plugins while preserving consistency through a Mode Isolator and self-distillation.
Empirical Validation: Demonstration that the framework achieves significant training acceleration (up to 4.0×) while retaining nearly 99% of the baseline performance across different MLLM architectures and resolutions.

4. Experimental Results

The authors evaluated DualSpeed on LLaVA-1.5 and LLaVA-NeXT using 9 representative visual understanding benchmarks (e.g., VQAv2, GQA, MME, MMBench).

Training Speedup:
- LLaVA-1.5: Achieved a 2.1× overall training speedup (with a 5.8× speedup in the pretraining phase).
- LLaVA-NeXT: Achieved a 4.0× overall training speedup (with a 5.3× speedup in pretraining), attributed to the higher proportion of visual tokens in high-resolution models.
Performance Retention:
- DualSpeed retained 99.61% of the baseline performance on LLaVA-1.5 and 99.04% on LLaVA-NeXT.
- In contrast, a naive application of pruning (NaivePrune) resulted in only ~95.89% performance retention, highlighting the severity of the training-inference gap.
Gap Closure: The framework successfully closed a performance gap of approximately 3.72% compared to naive pruning methods, bringing the model's performance nearly to the level of the full-training baseline.
Redundancy Analysis: Experiments indicated that up to 90% of visual tokens can be pruned during training without significant performance loss, provided the DualSpeed framework is used.

5. Significance and Claims

The paper claims that DualSpeed offers a novel pathway to scale MLLM training by addressing the computational bottleneck of visual tokens without sacrificing model capability. By decoupling the efficiency gains of pruning from the consistency requirements of inference, the framework enables:

Resource Efficiency: Drastic reduction in training time and computational costs.
Scalability: The ability to train high-resolution models (like LLaVA-NeXT) more feasibly.
Robustness: The resulting models are naturally compatible with both normal inference (full sequences) and pruning inference (pruned sequences), as the training process explicitly accounts for both data distributions.

The authors emphasize that their approach does not require architectural changes to the VTP methods themselves, allowing existing pruning techniques to be seamlessly integrated as plugins into the training pipeline.