OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models

Technical Summary: OmniFocus

Problem Statement
Omni-modal Large Language Models (OmniLLMs) have demonstrated the ability to jointly process audio and video, enabling tasks such as audio-visual understanding, empathetic reasoning, and captioning. However, audio-visual inputs generate extensive token sequences, leading to substantial inference costs that limit applicability in long-form settings and resource-constrained environments. While token compression is a standard solution for reducing inference costs in unimodal or vision-language models, applying it to OmniLLMs presents unique challenges: the method must simultaneously preserve audio evidence, visual evidence, and cross-modal alignment.

Existing audio-visual token compression methods, such as OmniZip, typically rely on unimodal guidance (e.g., using audio attention to guide the compression of both modalities). The authors identify two critical limitations in this approach:

1. Neglect of Temporal Locality: Unimodal-guided methods often fail to account for the temporal locality of query-relevant evidence, assuming that information density is uniformly distributed or aligned across modalities.
2. Modality Bias: By using a single modality as a guide, these methods implicitly assume that the guiding modality's information density distribution applies to the other. This leads to overfitting to the guiding modality while overlooking potentially critical evidence in the other, degrading performance on tasks requiring joint audio-visual reasoning or the non-guiding modality.

Methodology: OmniFocus
The authors propose OmniFocus, a training-free, query-guided token compression method designed to achieve modality-symmetric compression. Unlike previous approaches that use one modality to guide the other, OmniFocus performs independent importance estimation for video and audio based on the textual query. The method operates in three stages before the tokens are consumed by the LLM backbone:

1. Query-Aware Modality Scoring:

   • The method extracts query tokens from the LLM's embedding layer to ensure they share the same input space as audio and video tokens.
   • For each temporal chunk, it computes a relevance score for video and audio independently by measuring the maximum cosine similarity between the query representation and the tokens within that chunk.
   • This "max-similarity" approach captures sparse, query-matching evidence effectively, acknowledging that relevant information may be concentrated in specific tokens rather than distributed evenly.

2. Chunk-wise Budget Allocation:

   • Relevance scores are converted into compression scores (where higher relevance yields lower compression).
   • These scores are mapped to local drop ratios for each chunk under global modality-specific budget constraints.
   • A calibration step ensures the total retained token count matches the target global retention ratio (e.g., 25% or 35%) while allowing different temporal chunks and modalities to receive adaptive local budgets.

3. Dual-Score Token Selection:

   • Within each chunk and modality, tokens are selected based on two complementary criteria to preserve both cross-modal alignment and modality-specific salience:
     • Inter-modal Association: Tokens are scored by their similarity to the mean representation of the other modality in the same chunk. This preserves audio-visual correspondence.
     • Intra-modal Peak: Tokens are scored by their distinctiveness relative to the mean of their own modality. This preserves evidence that is unique to a specific modality and might be missed by cross-modal alignment alone.
   • The final selection splits the retained tokens equally between those with the highest association scores and those with the highest peak scores.

Key Contributions

1. Identification of Modality Bias: The paper empirically validates that unimodal-guided compression in OmniLLMs leads to modality bias, degrading performance on tasks requiring the non-guiding modality or joint reasoning.
2. OmniFocus Framework: The introduction of a training-free, query-guided method that performs symmetric compression. It estimates query relevance separately for audio and video, mitigating bias while preserving inter-modal associations and intra-modal peaks.
3. Empirical Validation: Extensive experiments across four audio-visual benchmarks (DailyOmni, WorldSense, OmniVideoBench, VideoMME) demonstrate that OmniFocus achieves a favorable accuracy-efficiency trade-off.

Results
Experiments were conducted on the Qwen2.5-Omni-3B and Qwen2.5-Omni-7B model families.

• Performance: At a 25% token retention ratio, OmniFocus outperformed existing baselines (OmniZip, DyCoke, and Random) on several major benchmark scores.
  • On DailyOmni (Qwen2.5-Omni-7B), OmniFocus achieved 59.40% accuracy, compared to 57.73% for OmniZip and 54.37% for DyCoke.
  • On WorldSense, OmniFocus achieved the best compressed results across all model-budget settings, outperforming OmniZip by up to 1.04 points on the 3B model.
  • On VideoMME, it achieved the best compressed average scores in all four settings.
• Efficiency: OmniFocus delivered significant speedups. On DailyOmni with the 7B model at 25% retention, it provided a 1.38× prefill speedup and 1.32× end-to-end speedup relative to the full-token baseline, while maintaining higher accuracy than other compressed methods.
• Ablation Studies: The authors confirmed that:
  • Max-similarity scoring outperforms average similarity.
  • Score-based adaptive allocation is superior to random or uniform allocation.
  • The hybrid inter-modal and intra-modal token selection strategy is critical; relying on only one leads to performance drops.
  • Modality-balanced scoring (considering both audio and video) is more robust than single-modality scoring.

Significance and Claims
The paper claims that query-guided modality-symmetric compression is a more reliable design choice for long-form audio-visual understanding than single-modality-guided token budgeting. The significance of OmniFocus lies in its ability to mitigate the modality bias inherent in previous asymmetric compression strategies without requiring model retraining. By preserving both cross-modal correspondence and modality-specific salient evidence, OmniFocus enables OmniLLMs to operate efficiently in resource-constrained environments while maintaining high fidelity in audio-visual reasoning tasks. The authors note that while the method is effective, its reliance on frozen input embeddings means it may be less precise for fine-grained events where initial token embeddings do not fully capture query intent, and its generalizability to architectures beyond the Qwen2.5-Omni family requires further evaluation.