💻 computer science

OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models

OmniPack is a training-free framework that enhances the efficiency of Omni-modal Large Language Models by combining pre-LLM structural redundancy removal with post-interaction semantic refinement, achieving superior performance-efficiency trade-offs across multiple benchmarks while significantly reducing computational costs.

Original authors: Wanshun Su, Yang Shi, Feihu Liu, Ziwen Yu, Yan Min, Zhuoran Zhang, Qixun Wang, Haotian Wang, Shixuan Liu, Yuanxing Zhang, Peng Wu, Chengfu Huo, Liang Ding

Published 2026-08-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wanshun Su, Yang Shi, Feihu Liu, Ziwen Yu, Yan Min, Zhuoran Zhang, Qixun Wang, Haotian Wang, Shixuan Liu, Yuanxing Zhang, Peng Wu, Chengfu Huo, Liang Ding

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to understand the world. You don't just give it a book to read; you show it a movie and play a soundtrack at the same time. This is the exciting world of "Omni-modal" AI, where computers learn to see, hear, and read all at once. But here's the catch: movies and soundtracks are huge. A single second of video or audio can be broken down into thousands of tiny digital "bricks" called tokens. If you try to feed a whole movie to the robot, it gets overwhelmed, like a student trying to read an entire library in one minute. It takes forever to process, costs a fortune in electricity, and often gets confused by the sheer volume of information. Scientists have been trying to solve this by throwing away the "boring" bricks before the robot even starts reading, hoping to keep only the important ones. However, the old ways of throwing things away were a bit clumsy—they sometimes tossed out crucial clues just because they looked different from their neighbors, or they tried to guess what was important before the robot had a chance to actually understand the story.

Enter OmniPack, a new, clever strategy designed to help these all-seeing, all-hearing robots work faster without losing their minds. Think of OmniPack as a two-step editing process for a chaotic movie script. First, before the robot even reads the script, OmniPack acts like a strict editor who scans the raw footage. It doesn't just delete the quiet parts; it looks for the "loud" moments (like a sudden crash or a bright flash) and also makes sure it doesn't skip over the quiet, important background scenery that happens far away in time. It groups similar bricks together so they don't take up extra space, creating a "highlight reel" that keeps the story's structure intact.

But the magic really happens in the second step. Once the robot has started reading the script and talking to itself, OmniPack steps in again. This time, it listens to the robot's questions. If the robot is asking, "What color was the car?", OmniPack knows to keep the tokens related to the car and the color, even if they were quiet or small. It uses the robot's own curiosity to refine the information, merging the most useful details together. The result is a massive reduction in the work the robot has to do. On a specific model called Qwen2.5-Omni-7B, OmniPack managed to shrink the work required to just 16.7% of the original amount while still keeping 98.0% of the robot's original smarts. Even when they pushed the limits and reduced the work to a tiny 6.8%, the robot still remembered 92.9% of what it was supposed to know. It's like shrinking a massive encyclopedia down to a single pamphlet, but somehow, the pamphlet still contains all the answers you need.

The researchers found that the old methods often failed because they tried to compress everything too early or too simply. They argued that throwing away tokens based only on how "loud" or "similar" they were often missed important, scattered clues. They also showed that trying to use audio to compress video (or vice versa) before the robot had a chance to mix the two senses together didn't work well. Instead, OmniPack suggests a "stage-specialized" approach: first, clean up the structure based on the specific type of media (video or audio), and then, after the robot has had a chance to mix them together, use the specific task or question to do the final, smart compression.

In their tests across five different challenge sets, OmniPack consistently beat every other method they compared it against. Whether they were testing on short clips or long videos, the new method always offered the best balance between speed and intelligence. The authors suggest that by coordinating these two stages—structural cleanup first, then smart, question-aware refinement—they can make these powerful AI models much more efficient without needing to retrain them from scratch. It's a way to make the super-smart robots faster and cheaper to run, keeping them sharp even when the information floodgates are wide open.

Technical Summary: OmniPack

Problem Statement
Omni-modal Large Language Models (Omni-LLMs) have achieved significant success in unified audio-visual understanding by jointly modeling text, images, videos, and audio. However, the dense tokenization required for visual and audio inputs results in sequences substantially longer than text, creating prohibitive computational and memory overhead during inference. While token compression is necessary for efficient deployment, existing methods face critical limitations under aggressive token budgets:

Inadequate Structural Modeling: Pre-LLM compression methods often rely on local importance or similarity, failing to capture long-range spatiotemporal structures. This leads to the loss of globally distributed evidence, such as sparse events in long videos or transient acoustic cues.
Insufficient Cross-Modal Collaboration: Existing approaches either compress before the LLM (where audio and visual representations have limited semantic interaction) or compress within the LLM (primarily relying on textual guidance without explicitly modeling audio-visual collaboration). Consequently, they fail to fully exploit task-relevant cross-modal evidence.

Methodology: OmniPack
The authors propose OmniPack, a training-free, two-stage progressive framework designed to coordinate structural compression before the LLM with semantic refinement within the LLM.

1. Modality-Specific Pre-LLM Compression
Operating before the LLM, this stage removes structural redundancy while preserving salient evidence and global coverage. It employs three distinct mechanisms:

Importance Selection: Identifies dominant tokens using encoder attention statistics and modality-specific structural cues. For video, this includes adjacent-frame variation and spatial distinctiveness; for audio, it includes adjacent-token variation to detect acoustic boundaries.
Coverage Selection: To prevent over-concentration on salient regions, this step selects additional representative tokens using a joint distance metric that combines feature similarity and normalized temporal/spatiotemporal distance. It utilizes DPC-KNN to ensure broad spatiotemporal coverage.
Similarity-Aware Token Merging: Instead of discarding unselected tokens, their information is aggregated into retained representatives. Unselected tokens are assigned to the most similar retained token based on feature similarity, positional proximity, and the importance of the target token, effectively transferring redundant information to the kept set.

2. Query-Conditioned Inner-LLM Compression
After sufficient multimodal interaction has occurred within the first $\ell$ Transformer blocks, a second compression stage refines the retained representations. This stage leverages:

Textual Guidance: A global query representation derived from text hidden states guides the selection of task-relevant tokens.
Audio-Visual Collaboration: The relevance score of a token is computed by considering its similarity to the text query, its similarity to the other modality's prototype (e.g., visual tokens considering audio context), and its within-modality representativeness.
Diversity-Aware Selection: Tokens are iteratively selected to balance relevance with semantic diversity, ensuring the retained set covers the necessary information spectrum.
Relevance-Modulated Merging: Unselected hidden states are incorporated into retained representatives via similarity-weighted averaging modulated by relevance scores.

Key Contributions

Identification of Limitations: The paper highlights that existing Omni-LLM compression fails under aggressive budgets due to inadequate long-range structure modeling and insufficient exploitation of query-conditioned cross-modal evidence.
Unified Framework: OmniPack introduces a training-free, progressive strategy that combines modality-specific pre-LLM compression (structural) with query-conditioned inner-LLM compression (semantic), merging removed information into retained representatives to minimize information loss.
Performance-Efficiency Trade-off: The framework is designed to enable aggressive token compression while maintaining high multimodal understanding performance.

Experimental Results
Extensive experiments were conducted on five benchmarks (AVUT, WorldSense, DailyOmni, VideoMME, LVOmniBench) using three Omni-LLM backbones (Qwen2.5-Omni-3B/7B and MiniCPM-o-2.6).

Performance: On Qwen2.5-Omni-7B, OmniPack preserves 98.0% of the original performance while reducing FLOPs to 16.7%. Under extreme compression (retaining only 6.8% of original FLOPs), it retains 92.9% of the original performance.
Comparison: OmniPack consistently outperforms state-of-the-art training-free baselines (including FastV-om, VisionZip-om, OmniZip, OmniSIFT, and SEATS variants) across diverse retention ratios.
Efficiency: At a 15%/7.5% retention ratio (pre-LLM/final), OmniPack achieves a 10.0× reduction in FLOPs and a 4.5× prefill speedup while preserving 95.6% of the original performance.
Ablation: Studies confirm that the combination of importance selection, coverage selection, and token merging yields the best pre-LLM results. Furthermore, inner-LLM compression benefits significantly from audio-visual collaboration and text-aware guidance compared to independent or generic strategies.

Significance
The paper claims that OmniPack addresses the critical bottleneck of computational overhead in Omni-LLMs without requiring model retraining. By shifting compression from purely structural cues to task-conditioned semantics through a coordinated two-stage process, the framework enables the practical deployment of Omni-LLMs on resource-constrained devices. The results demonstrate that aggressive token compression can be achieved without sacrificing the model's ability to understand complex, dynamic, and temporally evolving real-world scenarios, establishing a new state-of-the-art performance-efficiency trade-off in the field.

Technical Summary: OmniPack

More like this