💻 computer science

Adaptive Two-Stage Visual Token Pruning for Efficient Inference in Video-Language Models

This paper proposes a post-hoc, training-free two-stage adaptive token pruning strategy that first eliminates redundant frames and then dynamically adjusts token retention based on inter-frame correlations, achieving a 95% reduction in computation while improving video captioning accuracy by 7% at 10% token retention.

Original authors: Paribesh Regmi, Qingshuang Chen, Chi Zhang, Heba Aly, Yelin Kim, Hongda Mao

Published 2026-08-05

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Paribesh Regmi, Qingshuang Chen, Chi Zhang, Heba Aly, Yelin Kim, Hongda Mao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to understand the world. You give it a camera and a brain, and you ask it to look at a video and tell you what's happening. This is the world of "Vision-Language Models" (VLMs). Think of these models as a team: one part is the "eyes" (a vision encoder) that looks at pictures and breaks them down into tiny puzzle pieces called "tokens." The other part is the "brain" (a large language model) that reads those tokens and writes a story or answers a question.

The problem is that videos are huge. A single image can be broken into hundreds of puzzle pieces, and a video is just a stack of many images. If you try to feed a whole video to the robot's brain, it gets overwhelmed. It's like trying to read a thousand-page book in one second; the robot slows down, gets tired, and can't work on small devices like phones or security cameras. Scientists have tried to fix this by throwing away some puzzle pieces, but most of their methods are like using a cookie cutter: they cut out the same amount of pieces from every single video, whether the video is a boring, still shot of a wall or an action-packed car chase. This paper asks a better question: What if we could be smarter about what we throw away, cutting more when things are repetitive and less when things are exciting?

The Story of the Smart Video Cutter

The authors of this paper, a team from Amazon, noticed that current methods for speeding up video AI are a bit clumsy. They treat every video the same way, chopping off a fixed number of puzzle pieces (tokens) no matter what's actually happening on the screen. But videos are special; they have "temporal redundancy," which is a fancy way of saying that if you have a video of a cat sleeping, frame 10 looks almost exactly like frame 11, which looks like frame 12. Wasting the robot's brainpower on all those identical frames is silly.

To fix this, the team invented a "Two-Stage Adaptive Visual Token Pruning" strategy. Think of it as a two-step cleanup crew for a messy room.

Stage 1: The Frame Filter
First, the method looks at the whole video and asks, "Which frames are actually new?" If you have a video of a person walking, the first few frames might be identical. The algorithm acts like a selective editor, tossing out the boring, repeated frames entirely. It keeps only the most interesting moments, like a highlight reel. This is the "frame-level" pruning.

Stage 2: The Token Tamer
Now, the robot has a shorter video, but each remaining frame is still made of thousands of tiny tokens. Here is where the magic happens. Instead of cutting a fixed number of tokens (like "always keep 50%"), the method looks at the content of the video to decide how much to cut.

Imagine the tokens in a frame are a group of people talking. If everyone is saying the exact same thing (high redundancy), you only need to listen to one person to understand the group. But if everyone is saying something totally different (high diversity), you need to listen to everyone. The paper's method does exactly this: it analyzes the "correlation" between the tokens. It uses a mathematical trick called "eigen-decomposition" to measure how much the tokens are repeating each other.

If the video is static and repetitive (like a person rolling a ball down a ramp), the math shows a "steep decay," meaning the tokens are very similar. The system then says, "Okay, we can throw away a lot of these!" and keeps only a tiny fraction. But if the video is chaotic and dynamic (like a car chase with lots of camera movement), the math shows a "slow decay," meaning the tokens are all unique. The system says, "Hold on, we need to keep almost all of these!" and cuts very little.

The Results: Speed without the Stumble
The team tested this on several popular AI models, including LLaVA-Video, InternVL3, and Qwen2.5VL. They compared their smart, adaptive method against other "training-free" methods (methods that don't require retraining the AI from scratch).

The results were impressive. By using their two-stage approach, they could reduce the amount of math the computer had to do by a massive amount—up to 95%—while still keeping the AI smart. In fact, on a video captioning benchmark (where the AI describes what it sees), their method actually improved accuracy by 7% when they only kept 10% of the tokens.

To put that in perspective: if you have a video that usually takes a supercomputer an hour to process, this method could make it run in minutes, and the robot might even understand it better because it's focusing on the important parts instead of getting lost in the noise.

Why This Matters
The paper explicitly argues against the "one-size-fits-all" approach used by previous methods. They show that using a fixed ratio (like always keeping 30% of the data) is suboptimal because different videos need different amounts of data. Their method is "post-hoc," meaning it works on existing models without needing to retrain them, making it a plug-and-play upgrade for current technology.

The authors found that this adaptive strategy works consistently across different model sizes and types of videos. They even tested different mathematical ways to measure the "decay" of information and found that an exponential curve fit the data best, confirming that their way of measuring redundancy is the most accurate.

In short, this paper suggests that we don't need to build bigger, slower brains to understand videos. Instead, we just need to be smarter about what we feed them. By acting like a savvy editor who knows exactly when to cut the boring parts and when to keep the exciting ones, we can make video AI fast enough to run on everyday devices without losing its ability to see the world clearly.

Technical Summary: Adaptive Two-Stage Visual Token Pruning for Efficient Inference in Video-Language Models

Problem Statement

Vision-Language Models (VLMs) have demonstrated strong capabilities in image and video understanding but face significant inference latency due to the computational cost of processing thousands of visual tokens. This issue is particularly acute in video processing, where multiple frames must be analyzed simultaneously. For instance, a moderate 5-minute video clip can generate approximately 11,000 tokens, imposing a quadratic computational overhead on the Large Language Model (LLM) decoder that is prohibitive for resource-constrained edge devices.

Existing token reduction techniques suffer from two primary limitations:

Lack of Temporal Awareness: Most methods are designed for single-image inputs and fail to account for temporal redundancies across video frames.
Fixed Pruning Ratios: Current approaches typically apply a uniform, fixed pruning ratio across all inputs. This is suboptimal because the degree of redundancy varies significantly between different videos; a static ratio either wastes computation on redundant content or degrades performance by discarding critical information in complex scenes.

Methodology

The authors propose a novel two-stage adaptive token pruning strategy designed specifically for video inputs. This approach is entirely post-hoc, requiring no additional training or fine-tuning of the underlying VLM.

1. Two-Stage Pruning Pipeline

The method reduces redundancy hierarchically:

Stage 1 (Frame-Level/Temporal Pruning): The system first identifies and discards redundant frames. Using a diversity-based selection mechanism, it retains a subset of frames that capture the most informative visual content, effectively removing visually similar frames that contribute little new information.
Stage 2 (Token-Level/Spatial Pruning): Within the retained frames, the system performs token-level pruning to eliminate spatial redundancy.

2. Adaptive Retention Ratio

A core innovation is the dynamic determination of the token retention ratio ( $\beta$ ) in the second stage based on the specific content of each video. The process involves:

Correlation Analysis: The method analyzes the linear correlation structure of token embeddings within the retained frames.
Eigen-Decomposition: It constructs a correlation matrix of the token embeddings and performs eigen-decomposition. The resulting eigenvalues capture the variance explained by different components.
Decay Rate Estimation: The eigenvalue spectrum is fitted with an exponentially decaying function ( $\lambda_n = A e^{-\alpha n}$ $λ_{n} = A e^{- α n}$ ). The decay rate ( $\alpha$ $α$ ) serves as a quantitative measure of redundancy:
- A steep decay (high $\alpha$ ) indicates high redundancy (few dominant eigenvalues), suggesting fewer tokens are needed.
- A shallow decay (low $\alpha$ ) indicates high diversity, suggesting more tokens must be retained.
Ratio Calculation: The retention ratio is defined as inversely proportional to the decay rate ( $\beta = \kappa / \alpha$ ), where $\kappa$ is a scaling coefficient. This ensures that videos with high redundancy are pruned more aggressively, while complex videos retain more tokens.

Key Contributions

Two-Stage Pipeline: The design of a hierarchical pruning process that sequentially addresses temporal redundancy (frame selection) and spatial redundancy (token selection), outperforming single-stage token pruning.
Content-Dependent Adaptivity: The introduction of a data-driven mechanism to determine the pruning ratio based on token correlation, eliminating the need for manually chosen, "one-size-fits-all" ratios.
Training-Free Efficiency: The method achieves significant computational savings and accuracy improvements without requiring the training of additional compressor modules or fine-tuning of the VLM.

Experimental Results

The method was evaluated on the LLaVA-Video model (7B parameters) and extended to InternVL3 and Qwen2.5VL across five video-language benchmarks (VideoDC, VideoChatGPT, NextQA, PerceptionTest, Video-MME).

Performance Gains: The proposed method consistently outperformed state-of-the-art training-free baselines (including PruMerge, FastV, DivPrune, and LLaVA-Scissor).
- On the VideoDC captioning benchmark at 10% token retention, the method improved accuracy by +7% compared to baselines.
- On VideoChatGPT, it achieved a 2–4% improvement at low retention ratios.
Computational Efficiency: The method significantly reduced computational overhead. At 10% token retention, it achieved a ~95.5% reduction in TFLOPs while maintaining or improving accuracy.
Scalability: The approach demonstrated consistent performance gains across different VLM architectures and parameter sizes (ranging from 1B to 8B parameters).
Ablation Studies: Experiments confirmed that both the two-stage design and the adaptive retention ratio contribute to performance, with the full method outperforming variants that use single-stage pruning or fixed ratios. Furthermore, the exponential decay model for estimating redundancy was shown to be superior to linear or quadratic models.

Significance

The paper claims that this work addresses a critical bottleneck in deploying VLMs for real-time and edge applications. By explicitly modeling temporal redundancy and adapting pruning levels to the specific content of each video, the method achieves a superior trade-off between efficiency and semantic fidelity. The authors emphasize that their approach provides a principled way to estimate the necessary number of tokens for a given visual input, avoiding the inefficiencies of fixed-ratio strategies and the information loss associated with simple merging techniques. The results suggest that adaptive, content-aware pruning is essential for scaling video-language models to practical, resource-constrained environments.