💻 computer science

Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding

EviSelect is a fine-grained dynamic visual selection framework that leverages a target MLLM's internal attention evidence via sparse prefilling to guide an optimized stochastic policy, enabling adaptive spatiotemporal sampling that significantly reduces computational costs and accelerates long video understanding while maintaining superior performance.

Original authors: Bo Zhang, Wenxin Wang, Feng Chen, Zhihao Zhang, Zixuan Wang, Changsheng Li, Yinjie Lei

Published 2026-08-07

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Bo Zhang, Wenxin Wang, Feng Chen, Zhihao Zhang, Zixuan Wang, Changsheng Li, Yinjie Lei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to watch a movie and answer questions about it. This robot, known as a Multimodal Large Language Model (or MLLM for short), is incredibly talented at understanding pictures and words. However, there's a catch: movies are long, and robots have limited "short-term memory." If you feed the robot an entire two-hour film frame-by-frame, it gets overwhelmed, forgets the important parts, and wastes a massive amount of energy processing boring, repetitive scenes like a slow pan across a static wall. To fix this, scientists have been trying to teach the robot to be a better editor, picking only the most important moments to watch. But the old way of editing was like using a rigid, pre-written script: it relied on outside tools to guess what was interesting, often missing the subtle clues the robot actually needed to solve the mystery.

This is where a new approach called EviSelect comes in. Think of EviSelect not as a rigid editor, but as a curious detective that learns to read the robot's own mind. Instead of asking an outside expert what to look at, EviSelect peeks at the robot's internal "attention maps"—essentially, a heat map showing which parts of the video the robot's brain is naturally focusing on. By using a clever trick called "sparse prefilling" (which is like taking a quick, low-resolution snapshot of the whole movie to get a general vibe), EviSelect figures out exactly where the action is, how fast things are moving, and how much detail is needed. It then teaches a lightweight "selector" to make three smart decisions for every moment in the video: Should we keep this scene? How many frames should we show here? And how clear should the picture be?

The results are like magic for efficiency. In tests on three different long-video challenges, EviSelect managed to answer questions just as well as, or even better than, existing methods, but it did so by using about 50% fewer visual tokens (the digital building blocks of the video). This massive reduction in data didn't just save memory; it made the whole process 3.9 times faster. The paper suggests that by letting the robot's own internal evidence guide the selection, rather than relying on rigid rules or outside guessers, we can build video-understanding systems that are both incredibly smart and surprisingly efficient.

The Problem: The "Too Much Information" Bottleneck

Long videos are a nightmare for current AI. If you ask a robot to watch a 30-minute video and answer a question, it faces a dilemma. It can't remember everything, so it tries to pick out the "keyframes"—the most important moments. But the old methods for picking these frames were flawed. They were like a film editor who only knows how to cut scenes based on how loud the music is or how bright the colors are, ignoring the actual story. These methods used external tools (like a separate AI that scores similarity) to decide what to keep. The problem? That external tool doesn't know what the main robot is thinking. It might highlight a flashy explosion while the robot actually needed to see a quiet conversation to solve the puzzle.

Furthermore, these old methods were "rigid." They treated every video the same way, picking a fixed number of frames at a fixed size. This is like trying to read a book by looking at every single letter, even the spaces between words, regardless of whether the sentence is simple or complex. It wastes energy on boring parts and misses the nuance in the exciting parts.

The Solution: Reading the Robot's Mind

The authors of this paper, Bo Zhang and his team, proposed a new framework called EviSelect. Instead of guessing what the robot needs, EviSelect asks the robot directly.

Here is how it works, step-by-step:

The "Sparse Prefill" Trick: Before the robot watches the full video, EviSelect gives it a tiny, compressed version of the movie. It's like showing the robot a series of quick, blurry thumbnails of the whole film. This is cheap and fast.
The "Attention" Clues: Even though the video is blurry and short, the robot's brain still lights up in specific areas. EviSelect captures these "attention maps." It breaks them down into three types of clues:
- Relevance: Does this moment match the question?
- Time: How does this moment connect to the ones before and after it?
- Space: Is there a lot of detail needed here, or is it a simple background?
The Smart Selector: A small, lightweight AI (the "selector") looks at these clues and makes three dynamic decisions for every timestamp in the video:
- Keep or Drop: Should we watch this second at all?
- Sampling Rate: If we watch it, do we need 1 frame, 4 frames, or 8 frames per second? (Fast action needs more frames; a slow scene needs fewer).
- Resolution: Do we need a 4K picture, or will a low-res sketch do? (A clear face needs high res; a dark hallway might not).

The Training: Learning by "Group Comparison"

How do you teach a robot to make these split-second editing decisions? The authors used a technique called GRPO (Group Relative Policy Optimization). Imagine a game show where the robot tries to edit a video in eight different ways at once. The system then checks which version got the right answer. If a version got the answer right and used fewer pixels, it gets a huge reward. If it got the answer right but used too many pixels, it gets a smaller reward. If it got the answer wrong, it gets no reward, even if it was efficient.

This "group comparison" teaches the robot to find the perfect balance: getting the answer right while using the absolute minimum amount of visual data.

The Results: Faster, Smarter, Leaner

The paper tested EviSelect on three major benchmarks: MLVU, LongVideoBench, and Video-MME. These are like the "Olympics" for long-video understanding, featuring movies, surveillance footage, and complex narratives.

The findings were impressive:

Accuracy: EviSelect achieved state-of-the-art performance, beating existing methods like TSPO and Q-Frame. For example, on the Video-MME benchmark, it improved accuracy by 1.9% compared to the previous best.
Efficiency: It used only 1,701 visual tokens on average, whereas the previous best methods used around 3,360. That is a 49% reduction in data.
Speed: Because it processes less data, the end-to-end time dropped from nearly 10 seconds to just 2.55 seconds. That is a 3.9x speedup.

What This Means (and What It Doesn't)

The paper suggests that the key to understanding long videos isn't just throwing more computing power at the problem; it's about being smarter about what we look at. By using the robot's own internal "evidence" rather than external guessers, EviSelect adapts to the unique rhythm of every video.

However, the authors are careful to note the limits. Because EviSelect learns from the specific robot's internal attention, it might not work perfectly if you swap it onto a completely different type of robot without retraining. Also, this method requires access to the robot's internal "brain" (the attention maps), so it can't be used on closed-source "black box" models where that information is hidden.

In short, EviSelect proves that if you teach an AI to trust its own instincts about what matters, it can watch a movie, find the clues, and solve the mystery with half the effort and four times the speed.

Technical Summary: EviSelect – Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding

Problem Statement

Long-form video understanding using Multimodal Large Language Models (MLLMs) faces significant bottlenecks due to limited context windows and attention dilution. Existing approaches attempt to mitigate computational costs and context limits by selecting query-relevant frames. However, current methods suffer from three critical limitations:

Objective Misalignment: They rely on external proxy scorers (e.g., CLIP, DINOv2) and rigid heuristic rules. These proxies reflect semantic relevance but fail to align with the target MLLM's intrinsic evidence requirements, often missing subtle but critical visual cues.
Static Frame Retrieval: Most methods treat keyframe selection as an isolated problem rather than modeling a query-conditioned belief distribution over continuous temporal trajectories. This leads to temporally myopic selections that over-concentrate on salient segments while missing sparse, critical events.
Rigid Spatiotemporal Configuration: Existing paradigms typically adopt a "one-size-fits-all" approach, selecting a fixed number of frames at a fixed spatial resolution. This fails to accommodate the non-uniform spatiotemporal information density inherent in long videos, resulting in inefficient computational allocation.

Methodology: EviSelect

The authors propose EviSelect, a fine-grained dynamic visual selection framework that reformulates keyframe selection as a distribution-aware dynamic sampling policy driven by the target MLLM's internal evidence. The framework operates in three stages:

1. Evidence Grounding via Sparse Prefilling

Instead of using external proxies, EviSelect extracts endogenous evidence directly from the target MLLM's attention maps.

Sparse Prefilling: The video is partitioned into temporal segments with anchor frames. These anchors are highly compressed (spatially downsampled) and processed through the target MLLM using block-sparse attention. This allows for the efficient approximation of attention maps at a low computational cost.
Evidence Decomposition: The resulting attention maps are decomposed into three complementary cues without retraining or learnable parameters:
- Query-Frame Relevance ( $A_{qf}$ ): Guides the decision to retain or discard a timestamp.
- Cross-Frame Temporal Dependency ( $A_{ff}$ ): Guides the local sampling rate (how many frames to sample within a segment).
- Intra-Frame Spatial Saliency ( $A_{if}$ ): Guides the spatial resolution of sampled frames.

2. Distribution-Aware Dynamic Sampling

A lightweight selector (0.1B parameters) takes the decomposed evidence cues as input. It predicts three fine-grained policy distributions for each timestamp:

Retention Decision ( $z_t$ ): Whether to keep or discard the timestamp.
Local Sampling Rate ( $\rho_t$ ): The number of frames to uniformly sample from the corresponding temporal segment.
Spatial Resolution ( $\lambda_t$ ): The resolution to which the sampled frames are resized.
The joint policy is formulated as a product of these categorical distributions, enabling the generation of a multi-scale visual input ( $V_s$ ) tailored to the video's information density.

3. Policy Optimization with GRPO

The selector is optimized using Group Relative Policy Optimization (GRPO).

Reward Function: The reward combines a performance term (binary accuracy of the MLLM's answer) and an efficiency term (weighted combination of retention reduction and token cost reduction).
Optimization Strategy: To prevent the policy from simply minimizing cost at the expense of accuracy, the efficiency reward is activated only if the group of sampled candidates contains at least one correct answer. The policy is updated by maximizing the group-relative advantage, encouraging the allocation of computation dynamically according to the video's intrinsic information density.

Key Contributions

EviSelect Framework: A novel framework that reformulates keyframe selection as a distribution-aware dynamic sampling policy driven by the target MLLM's internal attention evidence, rather than external proxies.
Sparse Evidence Probing & Selector: A mechanism to rapidly approximate attention priors via sparse prefilling and a lightweight selector that predicts fine-grained spatiotemporal sampling policies (retention, rate, and resolution).
Efficiency and Performance: Extensive experiments demonstrate that the method achieves state-of-the-art performance while significantly reducing computational overhead.

Experimental Results

The method was evaluated on three benchmarks: MLVU, LongVideoBench, and Video-MME, using LLaVA-Video-7B and Qwen2.5-VL-7B as target MLLMs.

Performance: EviSelect achieved superior performance compared to existing methods (including TSPO and Q-Frame). For instance, on LLaVA-Video-7B, it improved accuracy by 1.9% on Video-MME, 0.9% on MLVU, and 0.7% on LongVideoBench compared to TSPO.
Efficiency: The method reduced the average number of selected visual tokens by approximately 50% (e.g., from 3,360 to 1,701 tokens for LLaVA-Video-7B) and achieved a 3.9× end-to-end speedup.
Ablation Studies:
- Components: Combining retention, adaptive frame sampling, and adaptive spatial resolution yielded the best results, confirming that orthogonal dimension reduction is effective.
- Evidence Source: Selectors trained on internal sparse-prefill priors significantly outperformed those trained on CLIP or random priors, validating the importance of MLLM-intrinsic evidence.
- Transferability: While the selector showed some cross-backbone transferability, performance was best when the selector and MLLM shared the same backbone, indicating partial backbone dependence.

Significance and Claims

The paper claims that EviSelect addresses the critical bottleneck of extracting decisive evidence from extremely long contexts by aligning the sampling strategy with the target MLLM's intrinsic requirements. By moving away from rigid heuristics and external proxies, the method dynamically allocates computational resources based on the actual information density of the video. The authors assert that this evidence-driven approach not only improves accuracy in long-form video understanding but also drastically reduces inference latency and token consumption, making long-video processing more feasible for current MLLM architectures.

Limitations Acknowledged: The authors note that because evidence priors are extracted from the internal attention of the target MLLM, generalization to other MLLMs is limited. Furthermore, the method is not applicable to closed-source MLLMs where internal attention mechanisms are inaccessible.