Technical Summary: O-MARC: Omni Memory-Augmented Compression Distillation for Efficient Video Understanding

1. Problem Statement

Omnimodal large language models (LLMs) have emerged as a promising direction for unified audio, visual, and language understanding. However, their inference remains costly due to the token-intensive nature of both audio and video modalities. Jointly processing video frames and audio streams generates long multimodal sequences, leading to substantial memory overhead and slow inference, particularly for long videos.

While token compression is a natural solution, extending it to audiovisual inputs is non-trivial. Unlike visual-only compression, audiovisual compression must handle two modalities with distinct redundancy patterns: video tokens often exhibit spatial and temporal redundancy, whereas audio tokens may encode brief but decisive cues. Crucially, the importance of each modality is often interdependent; a sound gains meaning when grounded in a visible scene, and a visual event may remain ambiguous without its corresponding audio. Existing compression methods often compress modalities in isolation, potentially removing the cross-modal evidence necessary for accurate reasoning.

Furthermore, existing benchmarks fail to fully isolate audio-visual association in noisy user-generated content (UGC). Many current benchmarks evaluate long video reasoning but rely predominantly on vision-based models that rarely utilize audio signals, or they do not explicitly filter out questions solvable by visual priors alone.

2. Methodology

The paper proposes a three-pronged approach: a new benchmark, a training-free compression method, and a compression-aware distillation framework.

2.1 UGC-AVQA Benchmark

To address the evaluation gap, the authors introduce UGC-AVQA, a public benchmark constructed from 1,000 public UGC short videos with 4,816 QA pairs.

• Focus: Fine-grained audio-visual association across events, scenes, and temporal changes.
• Construction: Videos are annotated with detailed captions and multiple-choice questions in four categories: Audio-Visual Event Progression, Audio-Visual Scene/Temporal Transition, Cross-Scene Alignment, and Fine-Grained Contrast.
• Difficulty Filter: To ensure questions require both modalities, an "audio removal test" is applied. A sample is retained for the benchmark only if a strong model (Gemini-3.1-Flash) fails when the audio track is removed. This filters out questions solvable by visual priors alone.
• Split: The dataset is split into a training set (794 videos, 3,168 QA pairs) and a benchmark set (206 videos, 1,648 QA pairs), with human review ensuring quality.

2.2 OMAC (Omni Memory-Augmented Compression)

OMAC is a training-free, plug-in compression method designed to preserve salient visual memory and temporally grounded audio anchors without modifying the backbone model.

• Visual Memory:
  • Frame Scoring: Frames are summarized and scored by query relevance. High-scoring frames are selected as key frames.
  • Token Selection: Within selected frames, tokens are scored by contrast against the frame centroid. Distinctive tokens are retained as explicit visual memory, while a compact frame memory token is formed via weighted pooling.
  • Structure: This creates a two-level memory structure: frame summaries (coarse memory) and distinctive local tokens plus a compact summary (fine-grained memory).
• Audio Memory & Budget Adjustment:
  • Anchors: Audio tokens are scored for importance. High-scoring tokens become "anchors," and nearby dropped tokens are merged into them to form compact acoustic memory slots.
  • Cross-Modal Budgeting: OMAC uses the visual memory distribution to adjust the audio budget. If a time segment retains significant visual memory, the audio budget for that segment is increased. Conversely, visually light intervals are compressed more aggressively. This ensures the compressed sequence preserves event structure across modalities rather than isolated tokens.

2.3 O-MARC (Omni Memory-Augmented Compression Distillation)

To make compact models robust to compressed inputs, the authors introduce O-MARC, a compression distillation framework based on Group Relative Policy Optimization (GRPO).

• Mechanism: For each query, the policy samples two rollouts: a "teacher" branch using full tokens and a "student" branch using OMAC-compressed tokens.
• Reward Shaping: The discrepancy between the teacher and student rewards ($\Delta_i$) measures compression-induced degradation. This discrepancy is used to reshape the GRPO advantage.
• Objective: The training objective is modified to bias policy optimization toward behaviors that remain robust under memory compression. Specifically, samples that are beneficial under GRPO but strongly degraded by compression receive larger positive updates. This allows the model to learn to reason effectively within a reduced token budget.

3. Key Contributions

1. UGC-AVQA: A new public benchmark for audio-visual association in UGC videos, featuring an audio-removal difficulty check to ensure questions genuinely require multimodal reasoning.
2. OMAC: A training-free plug-in compression method that preserves salient visual memory and uses visual context to dynamically allocate audio memory budgets.
3. O-MARC: A compression distillation framework that trains omnimodal models to reason robustly under compressed multimodal contexts, outperforming standard token pruning and contemporary training-based baselines.

4. Experimental Results

Experiments were conducted on models ranging from 3B to 30B parameters (Qwen2.5-Omni and Qwen3-Omni series) across four benchmarks: DailyOmni, UGC-AVQA, OmniVideo, and WorldSense.

• OMAC Performance: OMAC consistently outperforms direct token pruning (OmniZip) and full-token inference in efficiency-accuracy trade-offs.
  • On Qwen2.5-Omni-3B, OMAC improved the average score to 45.8, surpassing full-token inference (44.1) and OmniZip (41.0).
  • It reduced inference latency by 34.6% (1.53× speedup) and memory usage by 34.7% compared to full-token inference.
• O-MARC Performance:
  • Efficiency: On the 3B model, O-MARC (trained with only 3k samples) achieved an average score of 45.8, outperforming the full-token baseline (44.1) and OMAC alone (42.8).
  • Scalability: When scaled to the 7B model with 100k+ training samples, O-MARC achieved an average score of 51.1, outperforming the contemporary training-based baseline OmniSIFT (48.7) and significantly improving UGC-AVQA accuracy (64.6 vs. 56.4).
• Ablation Studies:
  • Compression Ratio: OMAC maintained high performance even at 70% compression, whereas OmniZip performance dropped significantly.
  • Guidance: The full OMAC design (Visual $\to$ Audio budget adjustment) outperformed variants where audio guided visual compression or where modalities were compressed independently.

5. Significance and Claims

The paper claims that efficient omnimodal reasoning should not be viewed solely as an inference acceleration technique but as a trainable condition.

• Interdependency: The results demonstrate that audio and video compression cannot be treated in isolation; preserving cross-modal evidence (e.g., using visual memory to guide audio budgeting) is critical for maintaining reasoning capabilities.
• Robustness: Through O-MARC, the authors show that compact models can be trained to reason effectively under reduced token budgets, often outperforming larger models or full-token inference in efficiency-constrained settings.
• Evaluation: The introduction of UGC-AVQA highlights a gap in current benchmarks, showing that reliable UGC audio-visual QA requires more than generic visual recognition or unimodal priors, necessitating explicit evaluation of audio-visual association.

The authors conclude that their unified study of benchmark construction, memory compression, and compression-aware training provides a pathway toward robust and efficient omnimodal reasoning for real-world, noisy video content.