Technical Summary: OmniSelect

Problem Statement

Omnimodal Large Language Models (OmniLLMs) have emerged to unify the understanding of audio, video, and text within a single autoregressive architecture. However, processing high-fidelity video and audio streams generates an excessive number of tokens, leading to quadratic complexity in attention mechanisms and creating significant computational and memory bottlenecks during inference.

While token compression has been explored for single-modal tasks (images, video, audio), existing solutions for OmniLLMs rely on fixed, modality-specific guidance. For instance, methods like OmniZip use audio anchors to guide video pruning, while others rely solely on visual cues. The authors argue that these "one-size-fits-all" strategies are inadequate because the relative importance of visual and auditory information varies significantly across different queries. As illustrated in the paper, some questions rely heavily on vision, others on audio, and some require a balanced integration. Using an inappropriate modality to guide compression can lead to the removal of critical information and erroneous model responses.

Methodology: OmniSelect

To address these limitations, the authors propose OmniSelect, a training-free, modality-adaptive token pruning framework. The method dynamically selects the appropriate compression strategy for multimodal inputs based on the specific query, avoiding fixed guidance mechanisms. The framework operates via a two-stage strategy:

1. Modality-Aware Dynamic Ratio Allocation

The first stage determines the global pruning strategy and allocates pruning ratios for each temporal group.

• Relevance Estimation: The system employs a lightweight AudioCLIP model to estimate cross-modal relevance. It computes similarity scores between the textual query and both visual and audio embeddings. To reduce computational cost, video frames are temporally downsampled, and the text is filtered to retain only nouns and adjectives.
• Strategy Classification: Based on the average similarity scores ($\bar{s}^{(v)}$ for video, $\bar{s}^{(a)}$ for audio) and a threshold $\theta$, the system categorizes the input into one of three pruning regimes:
  • Uniform Pruning: If $|\bar{s}^{(v)} - \bar{s}^{(a)}| \leq \theta$, both modalities are treated as equally important.
  • Video-Centric Pruning: If $\bar{s}^{(v)} > \bar{s}^{(a)}$, the strategy prioritizes video tokens.
  • Audio-Centric Pruning: If $\bar{s}^{(a)} > \bar{s}^{(v)}$, the strategy prioritizes audio tokens.
• Adaptive Allocation: Within the chosen regime, the system dynamically allocates pruning ratios ($\rho$) to each temporal group. It uses a sigmoid mapping based on the deviation of similarity scores from the mean, ensuring that groups with higher relevance to the query retain more tokens while satisfying a global pruning budget.

2. Temporal Group Pruning Pipeline (TGP2)

Once the strategy and ratios are determined, the system performs fine-grained token pruning within each temporal group:

• Audio Pruning: Salient audio tokens are selected using Top-K selection based on the attention matrix derived from the last layer of the audio encoder.
• Video Pruning: Salient video tokens are selected using Bottom-K selection based on the cosine similarity matrix of the vision embeddings. The authors choose Bottom-K (selecting tokens with the lowest similarity to others) over attention-based methods to avoid attention sink bias and reduce computational overhead, arguing that this effectively removes redundant, highly correlated content while preserving distinctive visual cues.

Key Contributions

The paper outlines three primary contributions:

1. Analysis of Fixed Guidance Limitations: The authors demonstrate that existing methods relying on static modality-specific guidance fail to account for the varying importance of modalities across different queries, leading to suboptimal compression.
2. OmniSelect Framework: They propose a training-free, modality-adaptive framework that dynamically selects between Uniform, Video-Centric, and Audio-Centric pruning strategies. It further performs fine-grained token pruning within temporal groups, adaptively allocating ratios to preserve informative tokens.
3. Empirical Validation: Extensive experiments on audio-visual benchmarks show that OmniSelect accelerates inference and reduces GPU memory overhead while maintaining high performance, achieving competitive results among training-free token compression methods.

Experimental Results

The authors evaluated OmniSelect on Qwen2.5-Omni models (3B and 7B parameters) across four benchmarks: WorldSense, DailyOmni, VideoMME, and OmniVideoBench.

• Performance Retention: OmniSelect retains 94% to 99% of the original full-token accuracy. In several instances (e.g., specific settings on OmniVideoBench and VideoMME), the method even surpassed the performance of full-token inference, suggesting that removing redundant tokens can improve prediction quality.
• Efficiency Gains:
  • Inference Speedup: Achieved a speedup of 1.19× to 1.33×.
  • Memory Reduction: Reduced GPU memory consumption by 2.58GB to 2.77GB.
  • Prefilling Time: Reduced prefilling time by 3.53× to 3.62× under 70% token compression.
• Comparison: OmniSelect consistently outperformed baselines like OmniZip (audio-guided) and DyCoke (temporal merging) across various frame budgets and compression ratios. The advantage was particularly pronounced in long-video scenarios and on the 3B model, which showed a more favorable efficiency–performance trade-off.
• Ablation Studies: Experiments confirmed that the dynamic selection of pruning strategies (switching between Video-Centric, Audio-Centric, and Uniform) is superior to relying on any single fixed strategy.

Significance and Claims

The paper positions OmniSelect as a necessary evolution in OmniLLM optimization. The authors claim that the primary significance of their work lies in shifting from static, modality-specific guidance to dynamic, query-aware strategy selection. By explicitly modeling modality preference and enabling the model to adapt its compression strategy to the specific semantic needs of a query, OmniSelect effectively avoids the pitfalls of uniform compression.

The authors emphasize that their method is training-free, making it immediately applicable to existing OmniLLMs without the need for retraining or fine-tuning. They conclude that while their current threshold-based selection mechanism is effective, the optimal strategy often exists within their proposed taxonomy, and the main limitation lies in the precision of the strategy selection rather than the representation capacity of the pruning itself. Future work is suggested to explore more robust models for modality importance estimation.