💬 NLP

AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression

The paper proposes AVOC, a retrieval-inspired token compression framework that enhances hour-level audio-video understanding in Omni-Modal LLMs by reframing token selection as a top- $K$ retrieval problem based on relevance, importance, and diversity, thereby achieving state-of-the-art performance on long-form benchmarks while maintaining robustness in hour-long contexts.

Original authors: Yijing Chen, Wenhui Tan, Xiaoyi Yu, Yuyue Wang, Xin Cheng, Kaisi Guan, Hao Jiang, Xiangyang Li, Guojie Zhu, Ruihua Song

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yijing Chen, Wenhui Tan, Xiaoyi Yu, Yuyue Wang, Xin Cheng, Kaisi Guan, Hao Jiang, Xiangyang Li, Guojie Zhu, Ruihua Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant library of movies and meeting recordings that go on for hours. You want to ask a super-smart AI assistant a specific question about what happened in one of these long videos, like, "How many people walked into the basement after that specific line of dialogue?"

The problem is that the AI's "brain" (its memory window) is too small to hold the entire video at once. If you try to feed it the whole thing, it chokes. If you try to shrink it by just picking random frames, you might miss the crucial moment. If you try to keep every single detail, you run out of space.

Enter AVOC: The Smart Librarian

The authors of this paper created a new system called AVOC. Think of AVOC as a highly skilled librarian who has to summarize a 1-hour movie into a tiny, 10-page cheat sheet for the AI, but with a very specific goal: the cheat sheet must contain only the information needed to answer your question.

Here is how AVOC works, using three simple rules borrowed from how search engines find the best results:

1. Relevance: "Does this match the question?"

Imagine you ask the librarian, "Who walked into the basement?"

Old way: The librarian might show you a page about the kitchen or the weather outside because those scenes were loud or colorful.
AVOC's way: AVOC looks at your question first. It scans the video and audio and says, "Okay, I need to find the part where people are talking about the basement." It highlights the specific moments in the video and the specific words in the audio that directly relate to your question. This is called Text-Guided Scoring.

2. Importance: "Is this interesting even without the question?"

Sometimes, your question is vague, or the answer depends on something the video shows but doesn't explicitly talk about.

The Analogy: Imagine you are looking for a specific person in a crowd. Even if you don't know their name, you might spot them because they are wearing a bright red hat (a unique visual cue) or because they are the only one dancing (a unique audio cue).
AVOC's way: AVOC checks if a moment is "important" on its own. It looks at how the video and audio talk to each other. If a person's face (video) matches a specific sound (audio), that moment gets a high "importance" score, even if your question didn't mention it. This ensures the AI doesn't miss hidden clues.

3. Diversity: "Don't show me the same thing twice!"

This is the trickiest part. If you have a scene where a character walks into a room, and then walks out, and then walks back in, a dumb system might pick all three moments because they all look similar. This wastes space.

The Analogy: Imagine you are packing a suitcase for a trip. You don't need to pack three pairs of the exact same red socks. You need one red pair, one blue pair, and one green pair to cover different needs.
AVOC's way: AVOC uses a special rule called Temporal-Aware Diversity. It says, "If I already picked a moment where someone walks into a room, I won't pick the next second where they do the exact same thing." However, if the same thing happens an hour later in the movie, AVOC will pick that one too, because it's a different event in time. This keeps the summary fresh and covers the whole timeline without repeating itself.

The Result: A Super-Smart Summary

By combining these three rules, AVOC takes a massive, hour-long video and audio stream and compresses it into a tiny, super-efficient "token" sequence. It throws away the boring, repetitive, or irrelevant parts and keeps only the "golden nuggets" of information.

What did they find?

It works better than anyone else: When tested on long videos (up to 90 minutes), AVOC answered questions much more accurately than other top models. It beat the second-best model by a significant margin (about 5 points higher on average).
It finds the "Needle in the Haystack": They tested if the AI could find a specific secret number hidden somewhere in a 1-hour video. AVOC could find it almost perfectly, even in videos that were an hour long, while other models started to fail as the videos got longer.
It's fast: Even though it's doing this complex sorting, it doesn't slow the AI down much. In fact, because it throws away so much useless data, the AI can actually process the video faster than before.

In short, AVOC teaches the AI how to be a better reader: it doesn't just read every word of a 500-page book; it learns to skim, highlight the important parts, and ignore the fluff, so it can answer your question perfectly.

Technical Summary: AVOC – Enhancing Hour-Level Audio-Video Understanding via Retrieval-Inspired Token Compression

1. Problem Statement

Multimodal Large Language Models (MLLMs) have achieved significant success in short-form audio-video tasks. However, extending these capabilities to hour-level long-form audio-video understanding faces two primary bottlenecks:

Context Window Limitations: The massive token sequences generated by long-duration video and high-sampling-rate audio exceed the context windows of current MLLMs.
Information Redundancy: Raw audio-video streams contain substantial redundancy. Naive reduction strategies (e.g., uniform sparse sampling) often miss critical short-lived events, while dense sampling exhausts context budgets, leading to sequence truncation and the dilution of key cues.

Existing compression methods for Omni-Modal LLMs often rely on asymmetric, unidirectional designs (e.g., using audio to prune video or vice versa). These approaches risk discarding critical information if the "guiding" modality provides weak or sparse signals, failing to capture the complex, interdependent nature of real-world multimodal content like movies or meetings.

2. Methodology: AVOC Framework

The authors propose AVOC, a framework that reframes multimodal token compression as a top-K retrieval problem. Given a fixed context budget $K$ and a large pool of candidate tokens, the goal is to retrieve a compact subset that best supports answering a user query. AVOC integrates three classical Information Retrieval (IR) criteria into a unified, learnable compression pipeline positioned between modality encoders and the LLM backbone.

2.1 Core Components

The compression module operates in three stages to generate and select tokens:

Relevance (Text-Guided Cross-Attention Scoring):
- Mechanism: Treats the user's text query as the search query and multimodal tokens as the candidate corpus.
- Implementation: Computes cross-attention scores between text embeddings and multimodal token embeddings. The relevance score for a token is the average attention logit received from all textual tokens.
- Goal: Prioritizes tokens pertinent to the specific user query.
Importance (Bidirectional Video-Audio Cross-Attention Scoring):
- Mechanism: Captures intrinsic informativeness independent of the query (query-agnostic), addressing cases where the query is sparse relative to rich audio-visual content.
- Implementation: Within each temporal block, computes bidirectional cross-attention between video and audio tokens. Tokens that interact strongly with the opposing modality receive higher scores.
- Goal: Ensures that semantically dense regions (e.g., a scene with both dialogue and action) are retained even if not explicitly queried.
Diversity (Temporal-Aware Maximal Marginal Relevance - TA-MMR):
- Mechanism: Prevents the selection of redundant, adjacent tokens while preserving temporally distinct but semantically similar events.
- Implementation: A re-ranking stage using a modified MMR objective. Unlike standard MMR, TA-MMR constrains the redundancy penalty to a local temporal window ( $W$ ). It penalizes similarity only among tokens within $[\tau_i - W, \tau_i + W]$ .
- Goal: Suppresses informationally repetitive adjacent tokens (e.g., static scenes) without suppressing recurring events that occur at different times.

2.2 Selection and Training

Budget Allocation: The total token budget $K$ is split into modality-specific budgets ( $K_{video}, K_{audio}$ ) to ensure balanced cross-modal representation. TA-MMR is applied independently within each modality.
Differentiable Training: To enable end-to-end training, the discrete top-K selection is approximated using Gumbel-Softmax with a Straight-Through Estimator. The TA-MMR re-ranking is disabled during training (due to its greedy, iterative nature) and activated only during inference.
Two-Stage Training:
1. Stage 1: Fine-tune the LLM backbone on data without the compression module to establish a stable gradient signal.
2. Stage 2: Jointly train the randomly initialized compression module and the LLM.

3. Key Contributions

Novel Perspective: Reframing multimodal token compression as a top-K retrieval problem, allowing the adaptation of classical IR principles (relevance, importance, diversity) to the audio-video domain.
Learnable Compression Module: The design of a unified module that instantiates three IR criteria via tailored mechanisms: text-guided cross-attention, bidirectional cross-modal attention, and Temporal-Aware MMR.
Omni-Modal Capability: Development of an Omni-Modal LLM (AVOC) capable of processing hour-level audio-video streams, achieving both holistic comprehension and fine-grained retrieval under tight context budgets.

4. Experimental Results

The authors evaluated AVOC on multiple long-form audio-video benchmarks and a custom Needle-in-a-Haystack (NIAH) task.

Benchmark Performance:
- OmniVideoBench: AVOC achieved 39.8% average accuracy on the (10, 30] min subset and 42.3% overall, surpassing the second-best model by 4.9 and 5.5 points respectively.
- LVOmniBench: AVOC achieved 41.3% on the Medium subset and 35.5% on the High subset, with an overall average of 40.3%, outperforming the second-best by 5.5 points.
- WorldSense: AVOC achieved 52.6% accuracy, a 1.7 point improvement over the second-best.
- Observation: The performance gains scale with video duration, indicating the method is particularly effective for ultra-long content where redundancy is highest.
Audio-Video Needle-in-a-Haystack (AV-NIAH):
- AVOC demonstrated robust retrieval capabilities for "needles" (secret keywords) hidden in videos up to 3600 seconds (1 hour) long.
- Unlike baselines (e.g., OmniZip) which showed significant accuracy degradation beyond 2000–3000 seconds, AVOC maintained high accuracy across the entire duration-depth grid.
Ablation Studies:
- Removing any single component (Relevance, Importance, or Diversity) resulted in performance degradation, confirming their complementary nature.
- The optimal modality budget ratio was found to be 2:1 (Video:Audio), reflecting the higher information density of visual cues in the tested benchmarks.
- TA-MMR hyperparameters ( $\lambda=0.15, W=3$ ) were critical; setting the window radius to infinity (degenerating to standard MMR) caused performance drops, validating the need for temporal awareness.
Efficiency:
- The compression module adds minimal overhead (approx. 1.8s for full retention).
- Aggressive compression (retention ratio $\rho=0.1$ ) reduced prefilling latency by nearly 9x (from 4.45s to 0.5s) compared to the uncompressed backbone.

5. Significance and Claims

The paper claims that AVOC represents a significant step toward Omni-Modal LLMs capable of reasoning over rich, hour-long multimodal content. By moving beyond rigid, asymmetric compression strategies to a symmetric, retrieval-inspired approach, AVOC addresses the fundamental trade-off between context window limitations and information redundancy. The authors posit that their framework enables models to maintain robust understanding and fine-grained retrieval capabilities in real-world scenarios (e.g., movies, meetings, tutorials) that were previously inaccessible due to sequence length constraints.