💻 computer science

When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding

The paper introduces EcoFrame, a training-free framework that enhances efficient long-video understanding by adaptively scheduling visual evidence through entropy-gated budget expansion and attention-guided candidate search, achieving superior accuracy-efficiency trade-offs compared to existing static and agent-based methods.

Original authors: Ke Li, Jiayu Chen, Maoliang Li, Zihao Zheng, Hailong Zou, Hengyi Zhang, Xuanzhe Liu, Xiang Chen

Published 2026-08-05

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Ke Li, Jiayu Chen, Maoliang Li, Zihao Zheng, Hailong Zou, Hengyi Zhang, Xuanzhe Liu, Xiang Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery, but instead of a single clue, you are handed a library containing millions of books. Your goal is to find the one paragraph that answers your specific question. If you try to read every single page, you'll run out of time and energy before you even finish the first chapter. This is the daily struggle for modern "Vision-Language Models" (VLMs)—super-smart computer programs that can see videos and answer questions about them. A typical long video might have tens of thousands of frames (individual pictures), but the computer's "brain" can only hold a tiny handful of them in its memory at once.

To solve this, scientists have tried two main tricks. The first is like a librarian who grabs a book from every tenth shelf, no matter what the question is. It's fast, but it often misses the crucial clue hidden on a random shelf. The second trick is like hiring a detective who reads a page, thinks hard, decides it's not enough, reads another page, thinks again, and repeats this process dozens of times. This is very accurate but incredibly slow and expensive. The big question in this field is: Can we get the detective's accuracy without the detective's slow, exhausting thinking process?

This paper introduces a new method called EcoFrame that answers "yes" by teaching the computer to listen to its own gut feelings. Instead of hiring a separate detective to decide what to look at, EcoFrame lets the main computer model use its own internal signals to figure out when it has seen enough and where to look next. The researchers found that when a model is confused, its "uncertainty" (measured as entropy) goes up, and when it knows the answer, its focus (measured as attention) gets sharp. By watching these signals, EcoFrame can stop early if the answer is obvious, or zoom in on specific parts of the video if the answer is tricky.

In their tests, EcoFrame proved to be a game-changer. On a popular test called Video-MME, it achieved an average accuracy of 64.4, beating the previous best relevance-based method, BOLT, which scored 63.5. Even more impressive, it was 1.85 times faster than those methods. When compared to the slow, detective-style "agent" methods, EcoFrame was up to 13.5 times faster while still getting the right answer just as often. The paper suggests that by using these internal signals, we can make video understanding both smart and speedy without needing expensive extra computing power.

The Problem: The "Too Much Video" Dilemma

Think of a long video as a massive, endless river of images. If you want to ask a computer, "What exercise is the man doing at the gym?" or "What is different about these three card tricks?", the computer can't possibly look at every single drop of water in that river. It has a limited "context window," which is like a small bucket it can carry. If the bucket is too small, it might miss the man lifting weights or the ace of spades.

Traditionally, computers used Uniform Sampling. Imagine you are walking along a river and you scoop up a cup of water every 10 meters. It's simple and fast, but if the interesting part happens between your scoops, you miss it completely. It doesn't matter if your question is easy or hard; you always scoop the same amount.

Then came Static Relevance Methods. These are smarter; they look at the whole river first, pick the "most interesting" cups of water based on the question, and bring those to the computer. But they are still rigid. They decide everything in one go. If the question is super hard and needs more clues, they can't go back and get more. If the question is easy, they still bring a full bucket, wasting time.

Finally, there are Agent-Based Methods. These are the detectives. They look at a few frames, ask the computer, "Is this enough?" If the computer says "I'm not sure," the agent goes back, finds new frames, and asks again. This is very accurate because it adapts to the difficulty of the question. But it's painfully slow because the computer has to "think" (reason) many times, like a student re-reading a textbook chapter over and over.

The Solution: EcoFrame's "Gut Feeling" System

The authors of this paper, Ke Li and their team, asked a simple question: Can the computer tell us when it's confused and where to look, without needing a separate detective to ask it?

They discovered that the computer actually gives away its secrets through two internal signals that are already there, for free:

Output Entropy (The "Confusion Meter"): When the computer generates an answer, it calculates how sure it is. If the answer is obvious, the computer is very confident, and its "entropy" (a measure of uncertainty) is low. If it's guessing, the entropy is high. EcoFrame uses this as a gatekeeper. If the entropy is low, the computer says, "I know this! Stop looking!" and gives the answer immediately. If the entropy is high, it says, "I need more clues," and asks for more frames.
Frame-Level Attention (The "Flashlight"): When the computer is looking at the video, it pays more attention to some frames than others. If it's staring intensely at a specific moment (like a magician's hand), that's a clue that the answer is nearby. EcoFrame uses this "flashlight" to decide where to search next. If the attention is focused, it zooms in on that specific area. If the attention is scattered, it spreads out to cover more ground.

How EcoFrame Works: The "Coarse-to-Fine" Hunt

Imagine you are looking for a lost key in a huge park.

Step 1: You start by looking at a few spots far apart (a low budget of frames).
Step 2: You ask your inner voice, "Do I see the key?" (Check the Entropy).
- If you feel confident (Low Entropy), you stop. You found it!
- If you feel lost (High Entropy), you need to look harder.
Step 3: You ask, "Where should I look next?" (Check the Attention).
- If your eyes were glued to a specific bush, you search that bush more closely (Local Search).
- If your eyes were wandering everywhere, you decide to check a new section of the park (Global Search).
Step 4: You pick the best new spots to look at, combining your "interesting spots" with "places you haven't checked yet" to make sure you don't miss anything.

This cycle repeats, but it's much faster than the detective method because the computer doesn't have to stop and write a report on why it's confused. It just uses the raw numbers from its own brain.

The Results: Fast, Smart, and Efficient

The team tested EcoFrame on three major video question-answering benchmarks: Video-MME, LongVideoBench, and MLVU. They used three different computer brains (VLM backbones): LLaVA-OneVision, Qwen2.5-VL, and InternVL-3.

The results were impressive:

Accuracy: On the Qwen2.5-VL model, EcoFrame achieved an average accuracy of 64.4. This beat the previous top relevance-based method, BOLT, which scored 63.5.
Speed: EcoFrame was 1.85 times faster than both AKS and BOLT.
Comparison to Detectives: When compared to the slow, agent-based method called A.I.R., EcoFrame was up to 13.5 times faster while maintaining similar accuracy.

The paper also showed that EcoFrame is "training-free," meaning it doesn't need to be retrained on new data to work; it just works with the existing computer models.

Why This Matters

The main takeaway is that we don't need to build expensive, slow "detective" agents to make video understanding smart. By simply listening to the computer's own signals of confusion and focus, we can make it adapt to the difficulty of the question on the fly. This means we can answer questions about long videos much faster, saving time and computing power, without sacrificing accuracy. It's a reminder that sometimes, the best way to solve a problem is to listen to the tool you already have.

Technical Summary: EcoFrame – Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding

1. Problem Statement

Long-video understanding requires Vision-Language Models (VLMs) to reason over videos containing tens of thousands of frames, which far exceeds the context windows of current models. Consequently, selecting a sparse subset of frames as visual evidence is critical for both inference efficiency and answer accuracy.

Existing approaches face a fundamental trade-off:

Static Relevance-Based Methods: These construct a fixed candidate pool and select a fixed number of frames (e.g., via CLIP scoring) in a single step. While efficient, they lack adaptivity; a fixed frame budget is often insufficient for complex queries or redundant for simple ones.
Agentic Methods: These use iterative reasoning and multi-round search to adaptively select frames. While highly adaptive, they incur significant scheduling overhead due to costly multi-round VLM reasoning and interactive search.

The core challenge is to achieve query-adaptive visual evidence scheduling—dynamically adjusting both the amount of evidence (frame budget) and the location of search (temporal regions)—without the high computational cost of agent-based iterative reasoning.

2. Methodology: EcoFrame

EcoFrame is a training-free framework that leverages internal signals from the VLM's own inference process to schedule visual evidence. It operates in a closed loop, progressively refining the evidence set based on two key observations derived from VLM behavior:

Output Entropy indicates evidence sufficiency.
Frame-Level Attention indicates where to search for additional evidence.

The framework consists of three main components:

A. Entropy-Gated Budget Scheduling

To determine when to stop or expand the frame budget, EcoFrame monitors the output entropy ( $e_{out}$ ) of the VLM's generated answer.

Mechanism: If the output entropy falls below a round-dependent threshold ( $\tau_t$ ), the current evidence is deemed sufficient, and the process terminates early. Otherwise, the frame budget is progressively expanded ( $B_{t+1} = \min(\lceil \alpha B_t \rceil, B_{max})$ ).
Relaxing Threshold: To account for the fact that difficult queries may naturally retain higher entropy even when answered correctly, the threshold is relaxed over rounds ( $\tau_t = \tau_1 + (t-1)\Delta\tau$ ). This prevents premature stopping on simple queries while avoiding unnecessary expansion on difficult ones.

B. Attention-Guided Candidate Proposal

To determine where to search for additional frames when evidence is insufficient, EcoFrame utilizes the VLM's frame-level attention scores.

Mechanism: The attention scores from the current sparse input set are propagated across the entire video timeline using "temporal cells" (intervals between adjacent input frames).
Search Prior: A candidate score is computed by multiplying the propagated attention prior (favoring high-attention regions) with a temporal diversity term (favoring regions far from existing frames).
- Concentrated Attention: Triggers dense local search around promising regions.
- Diffuse Attention: Triggers broader global coverage to ensure no temporal regions are missed.
Expansion: New candidates are selected from the expanded pool and their query-frame relevance scores are cached for the next round.

C. Marginal-Utility Frame Reselection

Once the candidate pool is expanded, EcoFrame reconstructs the input set $S_{t+1}$ for the next VLM inference.

Mechanism: Frames are selected based on a marginal utility score that balances query relevance (cached CLIP similarity) and temporal diversity (distance from already selected frames).
Benefit: This allows newly discovered high-relevance frames to replace weaker earlier frames while preventing temporal collapse (clustering of frames).

D. Initialization and Bounded Computation

The process starts with a uniformly sampled low-budget pool. The geometric expansion of the budget ensures that the cumulative VLM processing remains bounded (logarithmic in the number of rounds), and the candidate expansion is capped, ensuring overall computational efficiency.

3. Key Contributions

Training-Free Adaptivity: EcoFrame achieves query-adaptive evidence scheduling without requiring additional training or separate agent modules. It directly repurposes the VLM's internal inference signals (entropy and attention).
Dual-Signal Scheduling: It uniquely couples entropy (for stopping criteria/budget expansion) and attention (for spatial/temporal search guidance) to solve the coupled challenges of evidence sufficiency estimation and targeted search.
Efficiency-Accuracy Trade-off: By avoiding iterative agent reasoning, EcoFrame significantly reduces scheduling overhead while maintaining the adaptivity of closed-loop evidence acquisition.

4. Experimental Results

EcoFrame was evaluated on three major long-video benchmarks: Video-MME, LongVideoBench, and MLVU, across three distinct VLM backbones: LLaVA-OneVision, Qwen2.5-VL, and InternVL-3.

Accuracy vs. Efficiency: EcoFrame consistently outperforms static relevance-based methods (AKS, BOLT, FOCUS) and uniform sampling.
- On Qwen2.5-VL, EcoFrame achieved an average accuracy of 64.4%, surpassing the best static method (BOLT, 63.5%) while providing a 1.85× speedup over AKS and BOLT.
- Compared to the agent-based method A.I.R., EcoFrame maintained comparable accuracy while achieving up to a 13.5× inference speedup.
Ablation Studies:
- Removing entropy-gated scheduling (using fixed budgets) resulted in higher latency for simple queries or lower accuracy for complex ones.
- Removing attention-guided proposal (using fixed candidate pools) led to non-adaptive search and missed crucial evidence.
- Marginal-utility reselection was shown to be superior to selection based solely on relevance or coverage.
Latency Breakdown: The framework introduces minimal overhead for candidate scoring, with the majority of time saved by early exits on simple queries and reduced VLM inference rounds compared to agent-based loops.

5. Significance and Claims

The paper claims that EcoFrame demonstrates the value of reusing internal signals produced by the target VLM for lightweight, adaptive evidence scheduling. It addresses the central question of how a VLM can adapt its visual evidence to each query without expensive iterative reasoning.

The authors position EcoFrame as a solution that resolves the trade-off between the efficiency of static methods and the adaptivity of agentic methods. By leveraging output entropy and frame-level attention, EcoFrame achieves a superior accuracy-efficiency trade-off, making it a practical approach for efficient long-video understanding in resource-constrained environments. The code is made available to facilitate further research in adaptive visual evidence scheduling.