Technical Summary: OmniDrop

Problem Statement

Omni-modal Large Language Models (Omni-LLMs) have demonstrated significant potential in holistic multimodal understanding, integrating text, audio, and video. However, a critical bottleneck for real-time applications and long-form reasoning is the "token explosion" caused by high-resolution temporal data. Audio and video signals generate significantly more tokens than text (e.g., a one-minute video clip can project to over 10,000 tokens), leading to quadratic growth in computational and memory overheads.

Existing token compression methods for Omni-LLMs typically perform pruning at the input embedding level. These approaches rely on assumptions that audio-video tokens are semantically aligned via shared representation spaces or temporal co-occurrence. The authors argue these assumptions are unreliable for two primary reasons:

1. Weak Pre-LLM Alignment: Audio and video embeddings occupy distinct subspaces within the LLM embedding space before processing, resulting in low cross-modal cosine similarity that fails to identify meaningful semantic relationships.
2. Unreliable Temporal Co-occurrence: Simultaneous audio and visual events do not necessarily imply semantic correspondence (e.g., background music or off-screen narration may not align with visible content).

Consequently, input-level pruning based on embedding distance or co-occurrence often discards important modality-specific evidence, leading to suboptimal token selection.

Methodology: OmniDrop

To address these limitations, the authors propose OmniDrop, a training-free, layer-wise token pruning framework. Unlike prior methods, OmniDrop progressively prunes tokens within the LLM decoder layers rather than at the input stage, utilizing three core mechanisms:

1. Progressive Layer-wise Token Pruning (PLP)

Motivated by the observation that omni-modal information fusion occurs progressively through decoder layers, OmniDrop adopts a dynamic pruning schedule.

• Early Layers: Pruning is conservative (low ratio) to allow sufficient omni-modal information fusion and integration between audio and video streams.
• Deep Layers: Pruning becomes aggressive (high ratio) once the model has formed clear query-relevant token preferences.
• Implementation: The pruning ratio $p_l$ for layer $l$ follows a sigmoid function:
  $$p_l = p_{init} + (p_{final} - p_{init}) \cdot \sigma(l, t_{mid}, \beta)$$
  This ensures tokens are preserved in shallow layers for integration and removed in deeper layers where informative tokens are more distinguishable.

2. Query-Guided Token Importance

Instead of comparing audio and video tokens directly, OmniDrop uses the text query as a semantic mediator.

• Mechanism: The importance score $S$ for audio and video tokens is calculated based on the average attention scores from text tokens to the audiovisual tokens:
  $$S = \frac{1}{n_T} \sum_{q} \text{Attn}_{q \to \{V,A\}}$$
• Adaptivity: This allows the model to dynamically allocate token budgets based on task requirements. For audio-centric tasks, more audio tokens are retained; for video-centric tasks, more video tokens are kept. This ensures minimal information loss for the specific query.

3. Temporal Diversity Score (TDS)

To prevent the over-pruning of tokens that are temporally distant from the primary attention peak (which could lead to the loss of global context), OmniDrop introduces a diversity metric.

• Process:
  1. Identify the "key chunk" ($c_{max}$) containing the token with the highest importance score.
  2. Select a candidate buffer of the lowest-scoring tokens.
  3. Calculate a normalized temporal distance $D$ from $c_{max}$.
  4. Augment the final score: $S_{TDS} = S + \lambda_{div} \cdot D$.
• Effect: Tokens temporally far from the primary attention peak receive a score boost, encouraging balanced token survival across the sequence and preserving global temporal context.

4. Intra-Modality Pruning

Before feeding tokens into the LLM, OmniDrop performs independent intra-modality pruning to remove redundancy:

• Audio: Retains tokens with the highest attention scores from the last layer of the audio encoder.
• Video: Uses Temporal Token Merging (TTM) to filter consecutive redundant visual tokens based on cosine similarity.
• Result: This pre-processing reduces the initial token count to ~45% before the layer-wise pruning begins.

Key Contributions

1. Analysis of Input-Level Limitations: The paper demonstrates that input-level audio-video similarity provides a weak signal for token pruning due to insufficient alignment in pre-LLM embeddings and the unreliability of temporal co-occurrence as a proxy for semantic relevance.
2. OmniDrop Framework: A novel, training-free, layer-wise pruning framework that leverages text queries for modality-agnostic and task-adaptive token selection. It employs a progressive pruning schedule and a temporal diversity score to balance efficiency and context preservation.
3. Empirical Validation: Extensive experiments on diverse benchmarks (VideoMME, WorldSense, AVUT) show that OmniDrop outperforms existing SOTA training-free methods (OmniZip, DASH) while significantly reducing latency and memory usage.

Experimental Results

Evaluated on Qwen2.5-Omni (7B and 3B models), OmniDrop achieved the following:

• Performance Gains: Outperformed all baselines by up to 3.58 points (specifically on the AVUT benchmark with the 7B model). Notably, at 30% token retention, OmniDrop surpassed the 100% full-token baseline on VideoMME.
• Robustness: Under extreme compression (20% retention), OmniDrop showed minimal performance degradation (average drop of 0.18–0.47 points) compared to baselines which dropped by over 1.0 point.
• Efficiency: Reduced prefill latency by up to 40% and GPU memory usage by up to 14.7% compared to full-token inference, while maintaining competitive performance.
• Task Adaptivity: Analysis confirmed that the query-guided mechanism successfully adapts token retention ratios to specific task needs (e.g., retaining more audio for audio recognition tasks and more video for scene recognition).

Significance and Claims

The paper claims that OmniDrop offers a practical solution for scalable and responsive Omni-LLMs by fundamentally shifting token pruning from the input level to the decoder layers. By leveraging text queries as a guidance signal and acknowledging the progressive nature of omni-modal fusion, the method achieves superior efficiency-performance trade-offs without requiring model retraining or architectural modifications.

The authors acknowledge limitations, noting that the method currently depends on text prompts for token selection, restricting its use in scenarios with only audio/video inputs. Additionally, as a training-free framework, hyperparameters (such as pruning schedules) are determined empirically rather than learned. Future work aims to explore methods for capturing semantic relationships without textual guidance and optimizing pruning decisions via calibration data.