💻 computer science

DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression

The paper proposes DASH, a training-free framework that efficiently compresses omnimodal tokens by using audio embeddings to detect semantic boundaries and dynamically segment video tokens, thereby preserving critical information while significantly reducing inference costs without sacrificing accuracy.

Original authors: Bingzhou Li, Tao Huang

Published 2026-03-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bingzhou Li, Tao Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to summarize a two-hour movie for a friend who only has 10 minutes to listen.

If you used an old-school method, you might say: "I'll just cut out every 10th second of the movie, no matter what's happening."

The Problem: You might accidentally cut out the exact moment the hero pulls out a gun (a boring 10-second stretch of silence before the action) or chop a funny joke in half. You end up with a summary that makes no sense because you ignored what was actually happening and just followed a rigid clock.

DASH is a new, smarter way to summarize movies (or any video with sound) for AI models. It acts like a smart editor that understands the story, rather than just a clock.

Here is how DASH works, broken down into simple concepts:

1. The "Audio Anchor" (The Conductor)

In a movie, the audio (dialogue, music, sound effects) usually tells you when the scene is changing.

The Analogy: Think of the audio as the conductor of an orchestra. When the conductor stops waving the baton or changes the tempo, the musicians (the video) usually change what they are playing too.
How DASH uses it: Instead of looking at the video frames blindly, DASH listens to the audio first. It waits for the "conductor" to signal a change (like a pause in speech or a shift in topic). These signals tell DASH exactly where one "scene" ends and the next begins.

2. Dynamic Chunking (Cutting by Story, Not by Time)

Old methods chop videos into equal-sized blocks (like slicing a loaf of bread into identical 1-inch pieces).

The Analogy: Imagine slicing a loaf of bread where some slices are just crust (boring silence) and others are the delicious, dense center (exciting action). If you slice them all equally, you waste time on crust and miss the good stuff.
How DASH uses it: DASH cuts the video into variable-sized chunks based on the story.
- If the characters are having a long, boring argument, DASH makes a big chunk and summarizes it quickly.
- If a car chase starts or a new character enters, DASH makes a small, detailed chunk to make sure the AI sees every important detail.

3. The "Three-Alarm" System (Deciding What to Keep)

Once DASH has cut the video into story-based chunks, it needs to decide which specific frames to keep and which to throw away. Old methods only looked at one thing: "Which frame did the AI pay the most attention to?"

The Problem: Sometimes the AI gets distracted by a bright color or a random object and ignores the actual plot. If you only keep what the AI "looks at," you might miss the plot.
How DASH uses it: DASH uses a Three-Alarm System to pick the best frames:
1. The Boundary Alarm: "Is this the start of a new scene?" (Keep it! It's crucial for context).
2. The Uniqueness Alarm: "Is this frame totally different from the ones before it?" (Keep it! It adds new info).
3. The Attention Alarm: "Is the AI already looking at this?" (Keep it, but don't rely on it alone).

By combining these three, DASH ensures it never throws away a critical plot point just because the AI was momentarily distracted.

Why Does This Matter?

Speed: Because DASH is so smart about what to cut, it can throw away 75% of the video data (keeping only 25%) and still understand the movie perfectly.
Efficiency: This makes AI models run 3.5 times faster and use much less memory.
Smarter Results: It doesn't just save time; it actually understands the video better than older methods because it respects the natural flow of the story.

In short: DASH stops treating videos like a endless stream of random pixels. Instead, it treats them like a story, using the audio to find the chapters, and then carefully selecting the most important pages to read, saving time without losing the plot.

1. Problem Statement

Omnimodal Large Language Models (OmniLLMs) integrate audio and visual streams to perform complex tasks like video question answering. However, processing these streams generates massive token sequences (tens of thousands), leading to prohibitive computational costs due to the quadratic complexity of self-attention mechanisms.

Limitations of Existing Methods:
Current token compression techniques (e.g., fixed-window grouping, attention-based pruning) treat multimodal tokens as flat, uniformly structured sequences. They rely on:

Fixed Windows: Grouping tokens into rigid, equal-sized chunks regardless of semantic content.
Single-Signal Selection: Relying primarily on attention scores, which are often sparse and heavy-tailed.

Core Issue: These methods ignore the piecewise-coherent structure of audio-visual signals. Semantic transitions (e.g., topic shifts, speaker changes, scene cuts) often coincide with distributional shifts in embedding space. Aggressive pruning based on fixed windows or sparse attention fragments these semantic units, causing disproportionate information loss and degrading model performance, especially at high compression ratios.

2. Methodology: DASH Framework

The authors propose DASH (Dynamic Audio-driven Semantic cHunking), a training-free framework that aligns token compression with the intrinsic semantic structure of the data. It operates as a lightweight plug-in between modality encoders and the LLM.

The framework consists of three key stages:

A. Dynamic Semantic Chunking (Segment-Level)

Instead of fixed windows, DASH uses the audio stream as a semantic anchor to detect boundaries.

Mechanism: It computes the cosine similarity between adjacent audio embeddings. A sharp drop in similarity indicates a semantic discontinuity (e.g., a pause or topic shift).
Boundary Detection: A boundary is detected if the similarity drops below a threshold ( $\tau_a$ ) and the distance from the last boundary exceeds a minimum chunk size ( $C_{min}$ ) to avoid noise-induced fragmentation.
Result: This creates variable-length audio segments that respect natural prosodic and semantic units.

B. Audio-Driven Visual Segmentation (Cross-Modal Level)

To ensure cross-modal alignment without learnable parameters, audio boundaries are projected onto video tokens.

Temporal Ratio Mapping: Audio boundaries are linearly scaled to video token indices based on the ratio of total audio tokens to total video tokens.
Strength-Based Refinement: Projected boundaries are refined using a greedy strategy. Boundaries are accepted only if they result in video segments large enough for subsequent spatial-temporal compression (ensuring a minimum of 2 frames). Stronger boundaries (higher audio similarity drop) are prioritized.
Outcome: Both audio and video are segmented into shared, semantically coherent chunks.

C. Tri-Signal Fusion Token Scoring (Token-Level)

Within each dynamic segment, DASH selects tokens to retain using a fused importance score, addressing the sparsity of attention-only methods. The score is a weighted sum of three orthogonal signals:

Boundary Probability ( $w_b=0.4$ ): Prioritizes tokens at semantic transition points (structural criticality).
Probabilistic Density-Based Uniqueness ( $w_u=0.3$ ): Measures representational distinctiveness using a multi-scale Gaussian kernel on low-variance feature channels. This captures content uniqueness that attention might miss.
Attention Score ( $w_a=0.3$ ): Reflects the model's perceived salience.

Adaptive Compression: The retention ratio is adjusted per segment based on the audio retention rate. Information-dense segments (high audio retention) keep more video tokens, while sparse segments are compressed more aggressively. Additionally, frames at semantic boundaries receive a "protection boost" to preserve visual anchors for narrative transitions.

3. Key Contributions

Structure-Aware Formulation: The paper reframes token compression as a structure-aware segmentation problem, arguing that compression should follow semantic boundaries rather than positional regularity.
Training-Free Framework: DASH introduces a novel, parameter-free method that uses audio embeddings to detect semantic discontinuities and propagate them across modalities for explicit cross-modal alignment.
Tri-Signal Fusion: It proposes a robust scoring mechanism that fuses structural, content, and attention signals, mitigating the sparsity bias of attention-only selection and preserving transition-critical tokens.

4. Experimental Results

Experiments were conducted on AVUT, VideoMME, and WorldSense benchmarks using Qwen2.5-Omni (7B and 3B models).

Accuracy vs. Compression:
- DASH achieves superior accuracy at significantly lower token retention compared to state-of-the-art methods.
- At 25% token retention, DASH matches or exceeds the performance of prior methods (like OmniZip) operating at 35% retention.
- On WorldSense, DASH (25% retention) achieved 44.9% accuracy on the 7B model, surpassing OmniZip (44.7% at 35% retention) while using 30% fewer FLOPs.
Efficiency Gains:
- Prefill Speedup: DASH achieves up to 3.8× prefill speedup and 1.7× end-to-end latency reduction compared to full-token inference.
- Overhead: The boundary detection and scoring mechanisms add negligible computational overhead (<40ms).
Ablation Studies:
- Removing dynamic chunking or tri-signal fusion individually degrades performance, confirming that both structural segmentation and multi-signal scoring are necessary and complementary.
- Cosine similarity was proven to be the most effective metric for boundary detection compared to dot product or change rate.

5. Significance

DASH represents a paradigm shift in multimodal token compression. By recognizing that audio-visual signals possess an intrinsic, piecewise semantic structure, DASH moves away from rigid, position-based heuristics.

Efficiency: It enables aggressive token reduction (down to 25%) without sacrificing narrative continuity or cross-modal coherence, making real-time OmniLLM inference feasible on resource-constrained hardware.
Generalizability: As a training-free plug-in, it can be applied to various OmniLLM architectures without retraining, offering a practical solution for deploying large multimodal models in latency-sensitive applications.