💻 computer science

VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding

To address the saturation of conventional single-turn video understanding benchmarks, this paper introduces VideoGAIA, a rigorous, multi-turn, tool-augmented benchmark featuring 271 expert-verified tasks that reveals significant performance gaps in current frontier multimodal large language models and aims to drive the field toward agentic video understanding.

Original authors: Fan Zhang, Guangming Yao, Jinyang Wu, Hao Wu, Zheng Lian, Xinyu Geng, Jingdong Chen, Yi Yuan, Pheng-Ann Heng

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fan Zhang, Guangming Yao, Jinyang Wu, Hao Wu, Zheng Lian, Xinyu Geng, Jingdong Chen, Yi Yuan, Pheng-Ann Heng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

For years, the way we test artificial intelligence has relied on a simple game: show the computer a video, ask a single question, and wait for an answer. This approach, known as video understanding, has become a standard measure of how well machines can see and reason about the moving world. In recent times, the most advanced AI systems have become so proficient at this that they can answer nearly every question correctly, reaching a point where the test no longer reveals their true limits. It is as if a student has memorized the entire textbook and can recite any fact, yet we still do not know if they can actually solve a new problem in the real world. To move past this plateau, researchers needed a new way to measure intelligence, one that does not just ask for a fact but requires the machine to act like a curious human investigator.

A team of researchers from universities and technology companies in China has introduced such a new test, called VideoGAIA. Instead of asking a model to watch a video and answer a single question, this new benchmark treats the video as just the starting point of a much longer investigation. In this setup, the artificial intelligence acts as an agent, a digital assistant that must figure out what it does not know, go find that missing information, and then bring it all together to solve a complex task. The video might show a person holding a specific phone or a car driving through a city, but the answer to the question often lies outside the video entirely. The system must recognize a clue in the footage, decide to search the internet for more details, read the results of that search, and then return to the video to check if the new information fits. This process of looking, searching, reading, and checking happens over many turns, mimicking how a human would research a topic.

The researchers built this test by starting with a massive collection of one hundred thousand videos from the internet. They used powerful AI models to help them select interesting clips and generate questions that could not be answered by looking at the video alone. For example, a video might show a host in a studio displaying many phones, holding up a transparent-back silver handset beside a green product stand, and the question would ask for the specific chipset variant powering that model. The video itself does not show the chipset name or its marketing details. To answer correctly, the AI must first spot the phone in the video, then use a search tool to find the product's specifications online, read the details on a webpage, and finally confirm that the chipset matches the visual evidence. The team filtered these potential questions through a rigorous process, having human experts review every single task to ensure it was fair, difficult, and required genuine reasoning. After more than one hundred hours of human review, they settled on a final set of 271 tasks that cover six real-world areas, including technology, history, geography, daily life, culture, and economics.

When the researchers tested twenty of the most advanced AI models available today on this new benchmark, the results were stark. Even the most powerful systems, which had previously scored near ninety percent on older tests, struggled to reach sixty percent accuracy on VideoGAIA. The best-performing model, a system named Seed2.0-Pro, achieved an accuracy of 58.30 percent, while others fell significantly lower. This gap shows that while these machines are excellent at recognizing what is right in front of them, they are still learning how to use tools effectively to fill in the gaps. The study found that the biggest source of failure was not a lack of knowledge or an inability to read the web, but a failure to correctly identify the visual clues in the video in the first place. If the AI misidentified an object or a detail in the footage, it would search for the wrong information, leading to a wrong answer.

The researchers also observed that simply making the AI search more or read longer did not guarantee success. Some models made hundreds of tool calls, searching the web repeatedly, yet still failed to solve the task. Others made fewer calls but were more precise, stopping once they had gathered enough evidence. The most successful agents were those that could maintain a sense of uncertainty, realizing when they did not have enough information and knowing exactly what to look for next. They would revisit specific parts of the video to check a detail, cross-reference that with a webpage, and only then commit to an answer. This behavior is closer to how a human expert works, carefully verifying facts before drawing a conclusion.

The findings suggest that the next generation of artificial intelligence will not be defined by how well it can answer a question from a single image or video, but by how well it can navigate the world to find the answer. The VideoGAIA benchmark serves as a rigorous test for this new capability, revealing that the path to truly intelligent assistants requires more than just better vision or larger databases. It requires the ability to think, search, and verify in a continuous loop. While current models have made great strides, the fact that none of them could master this new test indicates that there is still a long way to go before machines can truly assist us in the complex, open-ended tasks of daily life. The work provides a clear roadmap for future development, showing that the key to progress lies in teaching these systems to be not just observers, but active investigators.

Technical Summary: VideoGAIA

Problem Statement

Video understanding has long served as a foundational task for evaluating Multimodal Large Language Models (MLLMs). However, the rapid advancement of foundation models has led to the saturation of conventional benchmarks. On leading single-turn video question-answering datasets like Video-MME, VideoMMMU, and MLVU, state-of-the-art models now achieve accuracies approaching 90%. This trend suggests that passive, single-turn video understanding has become a nearly solved problem, failing to adequately assess the capabilities required for next-generation AI assistants.

While multimodal agents are reshaping how AI systems interact with the world, the development and evaluation of video agents—systems capable of iteratively perceiving videos, invoking external tools, and synthesizing information to solve complex tasks—remain largely underexplored. Current benchmarks often rely on closed-world, single-turn protocols where models answer predefined questions without actively determining what evidence to inspect or consulting external resources. There is a critical need for a new evaluation paradigm that shifts from passive question answering to agentic video understanding, where video serves as evidence for accomplishing concrete, real-world tasks.

Methodology

The authors introduce VideoGAIA, a benchmark designed to evaluate general AI assistants in an agentic video understanding setting. The construction of VideoGAIA follows a rigorous multi-stage human-model co-design pipeline:

Data Collection: The process begins with a pool of 100,000 publicly available videos collected from the internet.
Task Construction Pipeline:
- Frame-Level Captioning: Videos are sampled at 1 FPS, divided into segments, and processed by frontier models (e.g., Gemini-3.1-Pro) to generate detailed visual descriptions and extract text via OCR.
- Evidence-Grounded Task Generation: Models (e.g., Claude-Opus-4.7, GPT-5.5) extract reusable "video anchors" (objects, text, logos, actions) and generate tasks that require identifying target attributes. Crucially, tasks are designed to be unsolvable by the video alone; they require retrieving external evidence via web search to connect visual anchors to specific facts.
- Quality Control: A strict multi-stage filtering process ensures task validity:
  - Rubric-Based QC: Tasks must require at least two video anchors, involve multi-hop reasoning (video $\to$ web), and conceal answers through visual referring expressions.
  - Cross-Modal Filtering: Tasks are tested to ensure they cannot be solved using only the video or only the web content, guaranteeing the necessity of multimodal reasoning.
  - Tool-Necessity Assessment: Tasks solvable by models using only video and world knowledge (without tools) are discarded.
  - Safety Screening: Removal of PII, sensitive data, and inappropriate content.
- Human Verification: The remaining candidate tasks undergo independent verification by three human experts. Only tasks with unanimous or majority approval (followed by adjudication) are retained.
Dataset Statistics: The final test set comprises 271 high-quality tasks spanning six real-world categories: Culture, Geography, Daily Life, History, Technology, and Economics. The dataset is divided into two difficulty levels: L1 (62 tasks) and L2 (209 tasks), with L2 emphasizing complex, multi-step evidence acquisition.
Evaluation Protocol: The authors evaluate 20 frontier MLLMs using a unified ReAct-style agent loop. Models are provided with coarse video frames and can interleave reasoning with three specific tools:
- Web Search: For retrieving external information.
- Page Visit: For converting webpages into textual content.
- Thinking with Videos: For revisiting and inspecting specific temporal segments of the video.
  Models are allowed up to 40 agent steps per task. Performance is measured by pass@1 accuracy against human-verified reference answers.

Key Contributions

The paper outlines three primary contributions:

Paradigm Shift: The work advocates for a transition from single-turn video understanding to agentic video understanding. This new paradigm treats video content as evidence for accomplishing concrete tasks rather than merely the object of a question, requiring agents to iteratively perceive, search, and synthesize information.
High-Quality Benchmark: VideoGAIA is a carefully curated benchmark containing 271 human-verified tasks. Its construction via a multi-stage human-model co-design pipeline ensures strong correctness, appropriate difficulty, and genuine requirements for multimodal reasoning and tool use.
Comprehensive Evaluation: The paper provides a systematic evaluation of 20 cutting-edge MLLMs under a unified agentic setting, revealing significant performance gaps compared to conventional benchmarks.

Results

The evaluation of 20 frontier models on VideoGAIA yields the following key findings:

Low Overall Accuracy: All evaluated models achieve accuracies below 60%. The top-performing model, Seed2.0-Pro, reaches 58.30%, followed by Qwen3.7-Plus (56.09%) and Kimi-K3 (54.61%). This is substantially lower than the ~90% accuracy observed on conventional benchmarks like Video-MME.
Difficulty Sensitivity: There is a consistent and significant performance drop from L1 to L2 tasks. For instance, Seed2.0-Pro drops from 75.81% on L1 to 53.11% on L2, indicating that L2 tasks successfully capture harder trajectories involving indirect references and multi-step evidence reconciliation.
Tool Usage Patterns: Analysis of tool usage reveals that web search dominates all models (60.5%–90.6% of calls). Interestingly, a higher volume of tool calls does not correlate with higher accuracy; for example, Gemini-3.1-Pro makes significantly more calls than GPT-5.5 but achieves lower accuracy.
Error Analysis: The primary source of error for all models is video perception (36.8%–48.0%), where incorrect visual anchoring contaminates downstream reasoning. Other bottlenecks include web retrieval (20.3%–31.0%) and evidence reasoning (14.2%–28.6%). Answer formulation errors are minimal (<3.3%).
Non-Monotonic Scaling: Generational improvements are not universal. While newer models in the Kimi, GLM, and Gemini families show improvements, newer Claude-Opus models (4.7 and 4.8) scored lower than their predecessor (4.6). Specifically, Claude-Opus-4.7 and 4.8 achieved 34.69% and 36.90% respectively, both falling below Claude-Opus-4.6's score of 47.97%. This suggests that general capability scaling does not automatically transfer to reliable multimodal tool use.

Significance and Claims

The authors claim that VideoGAIA serves as a timely and high-quality benchmark for evaluating the next generation of MLLMs. The paper argues that the current saturation of single-turn benchmarks masks the limitations of existing models in agentic video understanding.

The significance of VideoGAIA lies in its ability to expose the gap between models that can answer questions about videos and models that can use videos as evidence to solve real-world tasks. The results indicate that agentic video understanding is far from solved, presenting substantial opportunities for future research in model architectures, training strategies, and agent harnesses. The authors hope this benchmark will facilitate the transition from conventional video understanding toward a more robust, tool-augmented, and evidence-integrating paradigm for general AI assistants.