💬 NLP

ReGATE: Learning Faster and Better with Fewer Tokens in MLLMs

ReGATE is a teacher-student framework that accelerates multimodal large language model training by adaptively pruning redundant tokens using guidance losses and difficulty estimates, achieving faster convergence and superior performance while significantly reducing total token consumption.

Original authors: Chaoyu Li, Yogesh Kulkarni, Pooyan Fazli

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chaoyu Li, Yogesh Kulkarni, Pooyan Fazli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart robot to understand videos. Right now, the standard way to do this is like forcing the robot to read every single word in a script, even the boring ones like "the," "is," or "and," while it watches a movie. This is incredibly slow, expensive, and wastes a lot of energy because the robot is doing unnecessary work.

The paper introduces a new method called REGATE (Reference-Guided Adaptive Token Elision). Think of REGATE as a super-efficient study coach that helps the robot learn faster by telling it exactly which words to focus on and which ones to skip.

Here is how it works, using simple analogies:

1. The Problem: Reading the Whole Script

In the old way (Standard Training), the robot processes every "token" (a chunk of text) in a video description.

The Analogy: Imagine you are trying to learn a new language by reading a book where every single word is highlighted in bright neon. You spend hours staring at common words like "the" and "a," which don't actually teach you much about the story. You get tired, and you learn slowly.

2. The Solution: The "Teacher" and the "Student"

REGATE uses a two-person team:

The Student: This is the main robot model we are trying to train. It looks at both the video and the text.
The Teacher: This is a "frozen" (unchanging) version of the robot that only sees the text. It cannot see the video.

How they work together:
The Teacher looks at a sentence and asks, "Can I guess what this word means just by reading the text, without seeing the video?"

If the word is "the" or "is," the Teacher says, "Easy! I know this."
If the word is "red," "spinning," or "mixing," the Teacher says, "Whoa, I can't guess this without seeing the picture! This word needs the video."

3. The "Difficulty Score"

REGATE combines the Teacher's guess with the Student's own history.

The Analogy: Imagine the Student is taking a practice test. REGATE keeps a log of which questions the Student keeps getting wrong.
If the Teacher says, "You need the video for this word," AND the Student has been struggling with similar words before, REGATE marks that word as "High Priority."
If the Teacher says, "I know this word," and the Student has already mastered it, REGATE marks it as "Skip."

4. The Result: The "Smart Skip"

During training, REGATE creates a mask. It tells the robot: "Read these important words, but skip the boring ones."

The Analogy: Instead of reading the whole book page-by-page, the robot now only reads the highlighted sentences that actually move the story forward. It ignores the filler words.
The Benefit: The robot does 40% less work (processes fewer tokens) but learns just as well, or even better, because it isn't wasting energy on things it already knows.

What the Paper Claims (The Results)

The authors tested this on three different types of video-learning robots:

VideoChat2
VideoLLaMA2
InternVL3.5

They found that:

Speed: The robots reached their peak performance twice as fast as usual.
Efficiency: They used only 38% of the tokens (words/chunks) compared to the standard method.
Accuracy: In many cases, the robots trained with REGATE actually became smarter than the ones trained the old way, especially on complex video questions.
No Extra Cost: This method doesn't require building a new robot or adding extra parts; it just changes how the robot reads during training.

Summary

REGATE is like a smart filter for training AI. Instead of making the AI read every single word in a script, it uses a "text-only" expert to identify which words actually need the video context to be understood. By skipping the easy, redundant words, the AI learns faster, uses less electricity, and often ends up smarter than if it had read everything.

1. Problem Statement

Multimodal Large Language Models (MLLMs) face significant computational bottlenecks during training, primarily driven by the quadratic complexity of the self-attention mechanism relative to input sequence length. This issue is exacerbated in video tasks, where frames are tokenized into extremely long sequences.

Limitations of Existing Methods: Current efficiency strategies (token pruning, merging, compression) largely target inference using static or pre-scored tokens. They fail to adapt dynamically during the training phase.
Training Inefficiency: In standard training, the model processes every token in every forward and backward pass, regardless of whether the token is redundant or easily predictable from text alone. This wastes substantial compute resources and limits the ability to scale to longer contexts and larger datasets.
The Gap: There is a lack of principled, adaptive methods to estimate token importance during training that capture cross-modal dependencies without relying on heuristics (like random dropping) or introducing architectural changes.

2. Methodology: REGATE

The authors propose REGATE (Reference-Guided Adaptive Token Elision), a framework designed to accelerate MLLM training by dynamically selecting informative tokens while skipping redundant ones.

Core Architecture: Teacher-Student Framework

Student Model: The MLLM being trained (e.g., VideoChat2, VideoLLaMA2, InternVL3.5).
Teacher Model: A frozen, text-only version of the student's LLM backbone. It is created by removing the visual encoder and projector from the student model.
Mechanism: The teacher acts as a reference expert to estimate how much a specific token relies on visual context.

Token Scoring Mechanism

REGATE calculates a unified difficulty score ( $d_{b,i}$ ) for each token based on two complementary signals:

Reference Loss ( $\ell^{ref}_{b,i}$ ): The teacher model attempts to predict the token using only the textual context (with visual tokens replaced by placeholders).
- Low Loss: The token is predictable from text alone (redundant).
- High Loss: The token requires visual grounding (critical).
Student Historical Difficulty ( $m_{s,i}$ ): An Exponential Moving Average (EMA) of the student's cross-entropy loss for that token position over time. This tracks tokens that the student consistently struggles to learn.

The combined score is formulated as:
$d_{b,i} = m_{s,i} + \lambda \ell^{ref}_{b,i}$
Where $\lambda$ balances the static reference signal and the dynamic student learning signal.

Dynamic Token Gating & Sparsity

Mask Generation: For each batch, a binary mask is generated by selecting the top $k$ tokens with the highest combined difficulty scores.
Visual Token Preservation: Special visual tokens (e.g., <image>, <video>) are always retained to preserve multimodal structure, regardless of their difficulty score.
Dual-Cycle Schedule: To ensure stability, REGATE employs a deterministic schedule:
- Warm-up Phase: All tokens are processed for the first $F$ steps of every cycle.
- Sparse Phase: Only a fixed proportion ( $p_{sparse}$ ) of tokens are processed for the remaining steps.
Sparse Decoder Implementation: The transformer decoder is modified to perform self-attention and Feed-Forward Network (FFN) operations only on active (retained) tokens. Inactive tokens bypass computation but maintain residual connections to preserve gradient flow continuity. This is implemented efficiently using Flash Attention masks.

3. Key Contributions

Adaptive Training Pruning: Introduction of the first adaptive token pruning method specifically for MLLM training that dynamically identifies visually essential tokens without adding trainable parameters.
Model-Agnostic & Architecture-Free: REGATE integrates seamlessly into existing MLLMs (including those using LoRA or full fine-tuning) without modifying the model architecture or requiring additional learnable components.
Dual-Signal Guidance: A novel scoring mechanism combining a static cross-modal reference loss (from a text-only teacher) with dynamic student learning history, effectively isolating tokens that are both visually critical and challenging to learn.

4. Experimental Results

The authors evaluated REGATE on three representative MLLMs: VideoChat2 (Mistral-7B), VideoLLaMA2 (Qwen2-7B), and InternVL3.5 (Qwen3-14B) across 5 image and 8 video benchmarks.

Efficiency Gains

Token Reduction: REGATE achieves peak accuracy using only 38% of the tokens on average compared to standard training.
Training Speed: It reaches the baseline's peak accuracy 2x faster on the MVBench benchmark.
Resource Savings:
- VideoLLaMA2: Reduced training time from 129.6 to 64.0 GPU-hours (with aggressive pruning) while matching accuracy.
- InternVL3.5: Reduced token usage by 41.4% and training time by ~40% while surpassing baseline accuracy.
- Memory: Reduced average per-GPU memory usage by 7–8 GB (approx. 10–11%) across all models.

Performance Accuracy

Surpassing Baselines: With extended training, REGATE models consistently outperformed standard baselines across multiple benchmarks.
- Example: VideoLLaMA2-REGATE improved ScienceQA accuracy by 19.1% and MME by 33.1 points while using 41% fewer tokens.
- Example: InternVL3.5-REGATE achieved higher accuracy on Video-MME and MLVU benchmarks with significantly fewer tokens.
Generalization: The method proved effective across different backbones (Mistral, Qwen) and training strategies (Full Fine-tuning, LoRA).

5. Significance and Conclusion

REGATE addresses a critical bottleneck in the scaling of MLLMs: the computational cost of training on long multimodal sequences.

Paradigm Shift: It moves token efficiency from an inference-only concern to a training-time optimization, proving that models can learn better (higher accuracy) and faster by focusing computation on the most informative data points.
Practicality: By requiring no architectural changes and no extra trainable parameters, REGATE offers a drop-in solution for researchers and practitioners to train larger models on limited hardware.
Future Work: The authors suggest future directions include adaptive sparsity scheduling based on task complexity and exploring stronger teacher models for more complex reasoning tasks.

In summary, REGATE demonstrates that less is more in MLLM training: by intelligently skipping redundant tokens, models converge faster, consume less memory, and ultimately achieve superior multimodal understanding.