💻 computer science

Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extraction

This paper introduces VG-GUIBench, a new benchmark for evaluating MLLM-based GUI agents' ability to follow video tutorials, and proposes TASKER, a task-driven and scene-aware keyframe extraction algorithm that significantly improves performance on both VideoQA and video-guided agentic tasks.

Original authors: Sunqi Fan, Qingle Liu, Runqi Yin, Meng-Hao Guo, Shuojin Yang

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sunqi Fan, Qingle Liu, Runqi Yin, Meng-Hao Guo, Shuojin Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to learn how to fix a complex machine or play a new video game, but the only instruction manual you have is a three-hour long video.

If you try to watch the whole thing from start to finish, you'll get bored, and you'll likely miss the one crucial second where the mechanic shows you how to unscrew the bolt. If you just pick random seconds to look at, you might only see the mechanic walking around or staring at a wall, which doesn't help you solve the problem.

This paper introduces a new way to teach computers how to watch these long videos efficiently, and it does so in two main parts: a new "test" and a new "smart viewer."

Part 1: The New Test (VG-GUI-Bench)

The authors noticed that current tests for AI video understanding are like asking, "How many red cars did you see?" or "What color was the shirt?" These are simple, shallow questions. They don't test if the AI can actually learn a skill from a video and use it later.

So, they built a new test called VG-GUI-Bench.

The Analogy: Imagine showing an AI a video tutorial on "How to change a password on a phone app." Then, instead of asking a trivia question, you ask the AI to actually go into a different phone app and change the password for you.
The Goal: This tests if the AI can watch a video, understand the step-by-step procedure, and then act like a human agent to perform that same task on a computer screen.

Part 2: The Smart Viewer (TASKER)

The biggest problem with these tasks is that long videos are full of "fluff" (redundant frames) and the important parts are hidden in the middle. If you feed the AI too many frames, it gets confused. If you feed it too few, it misses the point.

The authors created a tool called TASKER (Task-driven And Scene-aware Keyframe searchER). Think of TASKER not as a camera, but as a very smart detective or a hiker with a map.

Here is how TASKER works, using a few analogies:

1. The "Tree" of the Video
Instead of watching the video linearly (second by second), TASKER treats the video like a tree.

It starts with the whole video as the trunk.
It splits the video into big chunks (branches).
It keeps splitting those chunks into smaller and smaller pieces until it finds the exact "leaves" (frames) that hold the answer.

2. The Two Types of "Smart" Search
TASKER uses a special AI brain (an MLLM) to decide which branch to explore next. It uses two different strategies, like a detective using two different clues:

The "What am I looking for?" Strategy (Task-Driven): The AI asks, "I need to find the part where the person types the password. Which part of the video looks most like that?" It zooms in on the most relevant section.
The "What changed?" Strategy (Scene-Aware): The AI asks, "Where did the scene change the most?" If the video goes from a kitchen to a living room, that's a big change. TASKER knows that big changes usually mean something important is happening, so it investigates those spots.

3. The "Stop When You Know" Rule
Most search algorithms run until they hit a hard limit. TASKER is smarter. It has an internal "confidence meter."

After looking at a few key frames, the AI asks itself: "Do I have enough information to answer the question?"
If it says "Yes, I'm 100% sure," it stops searching immediately.
If it says "I'm not sure yet," it digs deeper.
The Benefit: This means TASKER often finds the answer using only 15% of the video's frames, whereas other methods might waste time looking at 100% of the video.

The Results

When the authors tested this "Smart Detective" (TASKER) on both simple video questions and the complex "learn-a-skill" tasks (VG-GUI-Bench):

It got better scores than the previous best methods.
It did this while looking at far fewer frames, making it much faster and cheaper to run.

Summary

In short, this paper says: "Current AI video watchers are either too dumb (missing the point) or too slow (watching everything). We built a new test to see if AI can actually learn skills from videos, and we built a new 'Smart Detective' (TASKER) that knows exactly which parts of a video to look at to solve the problem, ignoring the boring stuff in between."

Technical Summary: Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extraction

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have achieved significant success on Video Question Answering (VideoQA) benchmarks, current evaluation paradigms largely focus on "low-level" perception, such as recognizing objects, attributes, and short-term actions. There is a critical gap in evaluating whether MLLMs can learn "high-level" procedural knowledge from video tutorials and generalize this knowledge to solve new, long-horizon agentic tasks.

Furthermore, a shared bottleneck exists across both VideoQA and video-guided agentic tasks: the effective extraction of task-relevant temporal content. Long videos often contain redundant segments, while key procedural evidence may be brief. Naive frame sampling strategies (e.g., uniform sampling) frequently miss critical moments or introduce excessive redundancy, degrading reasoning performance and computational efficiency.

2. Methodology

2.1 VG-GUI-Bench: A New Benchmark

To address the lack of evaluation for high-level video understanding, the authors introduce VG-GUI-Bench (Video-Guided GUI Benchmark).

Objective: Evaluate whether MLLM-based GUI agents can follow video tutorials to complete corresponding GUI interactive tasks.
Structure: Each entry pairs a tutorial video with a specific GUI task requiring similar procedural knowledge (e.g., navigating a UI to reset a password based on a video demonstration).
Action Space: The benchmark defines a standardized action space including CLICK, SCROLL, TYPE, PRESS (system keys), ZOOM, and FINISH.
Metrics:
- Accuracy: A weighted score (0.3 for correct action type, 0.7 for correct arguments).
- Completion: The proportion of correctly executed steps per episode.
- Efficiency: The average number of input frames consumed per prediction step.
- Performance Improvement Rate (PIR): Measures the gain in accuracy when video input is provided versus no video input.

2.2 TASKER: Task-driven And Scene-aware Keyframe searchER

The authors propose TASKER, a generalized keyframe extraction algorithm that frames the problem as a tree-structured graph search. Instead of pre-computing features for all frames, TASKER selectively extracts information during the search process.

Search Objective: Identify a sufficient set of keyframes whose combined information allows the model to answer a question or complete an agentic task.
Node Expansion: The video is initially split into $M$ segments (nodes). The algorithm iteratively selects a node to expand (subdivide) based on a cost function.
Cost Functions & Variants:
- TASKER-GBFS (Greedy Best-First Search): Task-Driven. The MLLM estimates the "distance" to the destination (the answer) by identifying missing visual information. It expands the segment most likely to contain this missing info.
- TASKER-Dijkstra: Scene-Aware. The MLLM identifies segments with the most significant scene changes (transitions in scenes, figures, or activities). These are prioritized as they are deemed more informative.
- TASKER-A*: Hybrid. Combines both task relevance ( $h(n)$ ) and scene dynamics ( $g(n)$ ), prioritizing segments that satisfy both criteria.
- TASKER-BFS: A naive variant that performs breadth-first expansion without MLLM-guided cost evaluation, used when MLLM overhead is a concern.
Termination & Confidence: Traditional deterministic termination is difficult for open-ended video tasks. TASKER employs a voting mechanism using two MLLM-based confidence estimators:
1. Self-Evaluation: The model reflects on its own answer and reasoning chain to output a confidence score.
2. Temporal Summarization: The model summarizes the captions of sampled frames into a cohesive video overview and predicts the answer again to output a second confidence score.
  The search terminates only when both scores exceed a threshold $C$ .
Frame Validation: Newly revealed frames are checked for visual redundancy and task relevance. Redundant frames are discarded, and segments yielding only redundant frames are added to a "frozen set" to prevent re-exploration.

3. Key Contributions

Taxonomy of Video Understanding: The authors propose a two-level hierarchy connecting low-level VideoQA (perception) with high-level Video-Guided Agentic Tasks (procedural learning and transfer), highlighting the role of "Video In-Context Learning."
VG-GUI-Bench: A new benchmark designed to systematically assess procedural knowledge transfer from video tutorials to GUI interactions, featuring 1,000 test cases with standardized action spaces.
TASKER Algorithm: A training-free, zero-shot keyframe extraction method that treats frame selection as a generalized graph search problem. It dynamically balances task relevance and scene dynamics to identify informative frames.

4. Experimental Results

VideoQA Benchmarks

TASKER was evaluated on EgoSchema (fullset) and NExT-QA.

Performance: Using GPT-4 as the base LLM, TASKER achieved 63.1% accuracy on EgoSchema (surpassing the best baseline, VideoTree, by 2.0%) and 77.4% on NExT-QA (surpassing the best baseline by 1.8%).
Efficiency: TASKER processes only visible frames. On EgoSchema, it achieved the same accuracy level (66%) as VideoTree while using only approximately 1/4 of the frames.
Generalization: The method consistently outperformed other baselines (including VideoAgent and VideoTree) even when using open-source MLLMs (Qwen3-VL).

VG-GUI-Bench Results

Baseline Comparison: On VG-GUI-Bench, TASKER-A* achieved the highest overall accuracy (40.96%) and Performance Improvement Rate (0.618) among keyframe selection methods, outperforming VideoTree and VideoAgent.
Task Completion: TASKER-Dijkstra achieved a completion rate of 74.39%, closely approaching the "Oracle Keyframe" upper bound (76.32%), demonstrating the ability to effectively deduplicate redundant frames while maintaining high task completion.
Efficiency: TASKER variants consistently required fewer frames per step compared to uniform sampling and other dynamic selection methods while achieving higher accuracy.

5. Significance and Claims

The paper claims that generalized keyframe extraction is a critical mechanism for bridging the gap between shallow VideoQA and complex, long-horizon agentic tasks. By formulating keyframe selection as a graph search problem driven by MLLMs, TASKER offers a training-free solution that improves both accuracy and frame efficiency.

The authors emphasize that their approach provides a unified mechanism for temporal information selection applicable to both reasoning (VideoQA) and execution (Agentic Tasks). The results suggest that effective video understanding relies not just on model capacity, but on the strategic selection of informative temporal content, a capability that TASKER addresses through its task-driven and scene-aware search strategy. The work highlights the potential of such methods to enable models to extract actionable knowledge from videos and apply it to downstream tasks without the need for extensive retraining.