💻 computer science

Enhancing Visual Token Representations for Video Large Language Models via Training-Free Spatial-Temporal Pooling and Gridding

The paper proposes ST-GridPool, a training-free method that enhances Video Large Language Models by integrating Pyramid Temporal Gridding and Norm-based Spatial Pooling to efficiently compress visual tokens while preserving critical spatiotemporal interactions.

Original authors: Bingjun Luo, Tony Wang, Hanqi Chen, Xinpeng Ding

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bingjun Luo, Tony Wang, Hanqi Chen, Xinpeng Ding

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to describe a long, action-packed movie to a friend who only has a very short attention span and can only remember a few key details. If you try to tell them every single frame of the movie, they will get overwhelmed and forget the important parts. If you just summarize it too quickly, you might miss the crucial plot twists.

This is exactly the problem video AI models face. They need to "watch" thousands of visual frames (tokens) to understand a video, but their "memory" (computational power) is limited. Current methods often try to shrink the video by simply averaging everything out, like taking a blurry photo of a whole crowd. This loses the important details.

The paper introduces a new, free upgrade called ST-GridPool. Think of it as a smart, training-free filter that helps the AI "watch" the video more effectively without needing to relearn how to see. It does this using two clever tricks:

1. The "Pyramid Temporal Gridding" (PTG) – The Time-Lapse Photographer

Imagine you are watching a video of a soccer match.

The Problem: Standard AI looks at the game in one uniform way. It might miss a quick hand gesture (a micro-movement) or fail to see the whole game strategy (a long-term trend) at the same time.
The Solution: PTG acts like a photographer who takes pictures at different speeds simultaneously.
- It creates a fine-grained view (looking at small chunks of 8 seconds) to catch quick actions like a player kicking a ball.
- It creates a coarse-grained view (looking at big chunks of 32 seconds) to understand the overall flow of the match.
- It then combines these different "scales" of time into a single, rich summary. It's like having a highlight reel that captures both the split-second goal and the full 90-minute strategy, all packed into a format the AI can easily digest.

2. The "Norm-based Spatial Pooling" (NSP) – The Spotlight Manager

Imagine a video frame where a person is talking in a busy, noisy room.

The Problem: Standard AI treats every part of the image equally. It gives the same attention to the speaker's face as it does to the boring, empty wall behind them. This wastes "memory" on the background.
The Solution: NSP acts like a spotlight manager. It looks at the "energy" (mathematically called the "norm") of every part of the image.
- It realizes that the speaker's face has high "energy" (lots of important information) and the wall has low "energy" (just background noise).
- It then amplifies the spotlight on the speaker and dims the lights on the wall.
- This ensures the AI focuses its limited memory on the most important parts of the scene, preserving the details that actually matter for answering questions.

The Result: A Smarter, Faster AI

The paper claims that by combining these two tricks, the AI becomes much better at understanding videos without needing any expensive retraining or changing its internal structure.

It's "Plug-and-Play": You can take an existing video AI (like LLaVA-Video) and just "snap" this method onto it. No new training required.
It Saves Money and Time: Because it focuses only on the important parts, the AI runs faster and uses less computer memory (GPU), especially for long videos.
It Works Better: In tests, this method helped the AI answer questions about long videos more accurately than before, beating other top methods even when the AI was forced to use very little "memory" (token budget).

In short, ST-GridPool is like giving a video AI a pair of smart glasses that automatically zoom in on the action and speed up the timeline, allowing it to understand complex stories in videos without getting tired or confused.

Technical Summary: ST-GridPool for Video LLMs

Problem Statement

Recent Multimodal Large Language Models (MLLMs) have advanced video understanding, yet they face a critical bottleneck: efficiently compressing visual tokens while preserving intricate spatiotemporal interactions. Existing Video LLMs, such as the LLaVA family, typically rely on simplistic 2D pooling or interpolation to reduce the quadratic complexity of self-attention mechanisms. However, these methods often treat all spatial and temporal tokens equally, overlooking the heterogeneous nature of video content. This leads to the loss of high-information regions (e.g., salient objects) and the failure to capture multi-grained temporal dynamics (ranging from rapid micro-motions to gradual scene evolution). Furthermore, most existing token compression strategies require costly retraining or specific architectural modifications, limiting their applicability as plug-and-play solutions.

Methodology: ST-GridPool

To address these limitations, the authors propose ST-GridPool, a novel, training-free visual token enhancement method designed specifically for Video LLMs. The approach consists of two synergistic components that refine visual tokens across temporal and spatial dimensions without altering model parameters or requiring retraining.

1. Pyramid Temporal Gridding (PTG)

PTG addresses the limitation of uniform temporal sampling by introducing a hierarchical gridding strategy.

Mechanism: The video sequence is partitioned into multiple layers, where each layer corresponds to segments of varying lengths ( $K_l = K \cdot 2^{l-1}$ ).
Process: For each segment, a subset of frames is uniformly sampled and spatially concatenated to form an intermediate token grid. This grid is resized back to the original resolution via bilinear interpolation to generate a "summary token."
Update: The last frame of each segment is updated with this summary token.
Benefit: This allows the model to capture both fine-grained (short-term) and coarse-grained (long-term) temporal dynamics simultaneously, enriching the temporal representation without additional trainable parameters.

2. Norm-based Spatial Pooling (NSP)

NSP addresses the issue of uniform spatial downsampling by leveraging the correlation between token norms and semantic richness.

Observation: Experiments on the HKU-IS dataset reveal a strong positive correlation between the L2 norm of visual tokens and the presence of salient objects. Background regions typically exhibit low norms, while high-information regions exhibit high norms.
Mechanism: NSP employs a dynamic 2D weighted pooling approach. Within a sliding window, the L2 norm of each token is computed and normalized using a softmax function to generate adaptive weights ( $\alpha_{m,n}$ ).
Process: The pooled token is calculated as a weighted sum of the visual features within the window, where weights are determined by the token's norm.
Benefit: This strategy selectively amplifies semantically rich regions (high-norm tokens) while suppressing low-information backgrounds, ensuring that critical visual details are preserved during compression.

Key Contributions

The paper outlines three primary contributions:

First Training-Free Video-Specific Enhancement: The authors propose the first visual token enhancement method specifically designed for Video LLMs that operates without retraining. It optimizes the compression process to improve performance while maintaining computational efficiency.
Novel Architectural Components:
- Pyramid Temporal Gridding: Introduces a hierarchical strategy to capture multi-grained spatiotemporal interactions across varying temporal lengths.
- Norm-based Spatial Pooling: Leverages the positive correlation between token norms and semantic importance to effectively preserve high-value visual information during compression.
Empirical Validation: Extensive experiments across six video understanding datasets demonstrate consistent performance improvements across multiple state-of-the-art models (LLaVA-Video, LLaVA-OneVision) and diverse tasks.

Experimental Results

The method was evaluated on widely adopted Video LLMs (LLaVA-OneVision-7B and LLaVA-Video-7B) across benchmarks for both Long Video Understanding (VideoMME, LongVideoBench, EgoSchema) and General Video Understanding (NexT-QA, TempCompass, MVBench).

Performance Gains: ST-GridPool achieved consistent improvements across all datasets. For instance, on LLaVA-Video-7B, it improved scores by +0.9% on VideoMME, +1.9% on LongVideoBench, and +1.2% on MVBench.
Token Reduction Efficiency: In comparisons with leading token reduction methods under strict token budgets (30% and 50%), ST-GridPool outperformed baselines like FastV, PruMerge, and FrameFusion, particularly under the stricter 30% budget where it achieved the best scores on all three long-video benchmarks.
Computational Efficiency: The method demonstrated a dual optimization, significantly reducing both inference time and peak GPU memory usage compared to the baseline, with savings becoming more pronounced as the number of input frames increased.
Ablation Studies: Removing either PTG or NSP resulted in performance drops, confirming that both components are necessary and complementary. The optimal hyperparameters were found to be a temperature $\beta=1$ and an L2-norm ( $p=2$ ).

Significance and Claims

The paper positions ST-GridPool as a scalable, "plug-and-play" solution that bridges the gap between efficient token compression and the preservation of complex spatiotemporal interactions. The authors claim that their method offers a powerful paradigm for enhancing visual token representations without the need for costly architectural modifications or retraining. By maximizing the preservation of semantically meaningful details through norm-based weighting and hierarchical temporal modeling, ST-GridPool enables existing Video LLMs to achieve more robust and accurate video understanding, particularly in scenarios requiring the analysis of long-range dependencies and dynamic activities. The code is made available to facilitate further research and adoption.