AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understanding

1. Problem Statement

Multimodal Large Language Models (MLLMs) struggle with long video understanding (e.g., hour-long inputs) due to two primary constraints:

1. Memory and Context Limits: Processing thousands of frames generates a massive number of visual tokens, quickly exhausting context windows and incurring prohibitive memory/compute costs.
2. Inefficiency of Existing Selection Methods:
   • Frame-level selection: Retains irrelevant background regions within selected frames, diluting useful signals.
   • Token-level selection: Often operates locally (within short clips), lacking a global mechanism to compare relevance across distant video segments.
   • Lack of Early Stopping: Current methods process all sampled frames even when the model has gathered sufficient evidence to answer the query, leading to unnecessary computation.

2. Methodology: AdaptToken

AdaptToken is a training-free, model-agnostic framework that dynamically selects informative visual tokens based on the MLLM's own self-uncertainty. It operates in four key stages:

A. Video Grouping and Processing

The long video is split into $G$ groups of frames ($F_1, \dots, F_G$). To ensure diversity, groups are processed in a "maximum margin" order (e.g., $F_1, F_{G/2}, F_{G/4}, \dots$) rather than strictly sequentially, allowing the model to sample the video broadly before committing to a full pass.

B. Group Relevance via Response Entropy

Instead of using external heuristics, AdaptToken uses the MLLM's response entropy as a signal for group relevance.

• Mechanism: For each group, the model generates a response to the prompt. The entropy ($e_i$) of the predicted token distribution is calculated.
• Logic: Low entropy (high certainty) implies the model has found strong evidence in that group. High entropy suggests the group is irrelevant or the model is guessing.
• Global Allocation: The certainty scores ($C_g$) of all groups are used to allocate a global token budget ($B$). Groups with higher certainty receive a larger share of the token budget via a softmax-weighted distribution:
  $$B_g = B \times \text{Softmax}(\{C_1, \dots, C_G\}/\tau)_g$$

C. Intra-Group Token Selection

Within each group, visual tokens are ranked using cross-modal attention from a specific layer of the MLLM (typically a late layer with strong retrieval capabilities).

• The attention weights between text queries and visual keys are aggregated to produce token relevance scores ($R_g$).
• The top-$B_g$ tokens are selected for each group based on these scores.

D. Location-Aware Global Token Removal

To prevent redundancy (e.g., selecting similar tokens from adjacent groups), a global token removal step is applied:

• It computes pairwise cosine similarity of token features and a temporal similarity term based on frame indices.
• Tokens with the highest mutual similarity are iteratively discarded until the target budget is met, ensuring diversity and temporal coverage.

E. Early Stopping (AdaptToken-Lite)

A variant of the framework introduces early stopping to improve efficiency:

• Trigger: If the model achieves high certainty (low entropy) on a sufficient number of groups (e.g., 3 groups with entropy $< 0.75$), the process halts.
• Result: Remaining groups are skipped, and the final token set is constructed only from the processed groups.

3. Key Contributions

1. Global Token Selection: Proposes a principled, training-free mechanism to compare relevance across distant video clips using model response entropy, solving the "local-only" limitation of prior token selection methods.
2. Entropy-Driven Early Stopping: Introduces AdaptToken-Lite, which dynamically stops processing once sufficient evidence is gathered, reducing inference time by ~50% without sacrificing accuracy.
3. Scalability and Robustness: Demonstrates effectiveness across diverse MLLM architectures (7B to 72B parameters) and scales to extremely long inputs (up to 10,000 frames), a regime where prior methods often fail or only report "Needle-in-a-Haystack" metrics rather than task accuracy.

4. Experimental Results

The method was evaluated on four challenging long-video benchmarks: VideoMME, MLVU, LongVideoBench, and LVBench.

• Performance Gains:
  • On Qwen2.5-VL 7B, AdaptToken achieved an average improvement of +6.7 over the baseline, outperforming state-of-the-art methods (e.g., FlexSelect, AdaReTAKE) on all benchmarks.
  • On InternVL2.5 8B, it improved the baseline by +5.6 on average.
  • On Qwen2.5-VL 72B, it maintained consistent gains (+5.7 average), proving scalability to larger models.
• Efficiency (AdaptToken-Lite):
  • Reduced average inference time by approximately 50% (e.g., from 17.8s to 8.6s on VideoMME) while maintaining comparable or slightly better accuracy than the full AdaptToken.
• Long-Context Scaling:
  • Successfully processed up to 10,000 frames, showing continued performance improvements on benchmarks with ultra-long videos (like LVBench), whereas prior methods typically plateau or degrade.
• Ablation Studies: Confirmed that both entropy-guided global allocation and location-aware token removal are critical components for maximizing performance.

5. Significance

AdaptToken represents a significant shift in long-video understanding by moving from static, heuristic-based selection to dynamic, model-aware selection.

• Training-Free: It requires no fine-tuning of the MLLM, making it immediately applicable to existing models.
• Efficiency: It addresses the computational bottleneck of long-context processing, enabling MLLMs to handle hour-long videos with standard hardware constraints.
• Generalizability: Its success across different model sizes (7B–72B) and architectures suggests that leveraging model uncertainty as a control signal is a robust strategy for multimodal reasoning.

In summary, AdaptToken enables MLLMs to "read" long videos more efficiently by focusing computational resources only on the most relevant moments, guided by the model's own confidence in its understanding.