💻 computer science

VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning?

This paper introduces VideoReasonBench, a novel benchmark designed to evaluate vision-centric, complex video reasoning by requiring models to track fine-grained operations on latent states, revealing that while most current multimodal LLMs struggle with such tasks, extended chain-of-thought reasoning significantly improves performance.

Original authors: Yuanxin Liu, Kun Ouyang, Haoning Wu, Yi Liu, Lin Sui, Xinhao Li, Yan Zhong, Y. Charles, Xinyu Zhou, Xu Sun

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuanxin Liu, Kun Ouyang, Haoning Wu, Yi Liu, Lin Sui, Xinhao Li, Yan Zhong, Y. Charles, Xinyu Zhou, Xu Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a magic trick where a magician shuffles a deck of cards, but they cover the cards with a cloth halfway through. You can see the hands moving, but you can't see the cards themselves. To know what the final order of the cards is, you have to remember exactly how the hands moved, keep a mental map of where every card was, and then calculate where they ended up.

This is exactly the kind of mental gymnastics that a new benchmark called VideoReasonBench is testing.

Here is a simple breakdown of the paper, using everyday analogies:

1. The Problem: The "Short Attention Span" of AI

Right now, AI models (the "brains" behind chatbots and image generators) are getting really good at looking at videos and answering simple questions like, "Is the dog running?" or "What color is the car?"

However, researchers found that these AIs are terrible at complex reasoning. If you ask them to track a sequence of events where things change, disappear, or hide, they often get lost. It's like asking a student who only memorized the first page of a book to solve a mystery that requires knowing the plot of the last 100 pages.

Existing tests were too easy. They were like asking, "Did the car turn left?" The AI could answer that instantly without really "thinking." The researchers wanted a test that forced the AI to actually use its brain.

2. The Solution: The "Video Puzzle Gym"

The team created VideoReasonBench, which is like a high-level gym for AI video understanding. Instead of just watching a video, the AI has to solve puzzles where the "answer" is hidden.

They designed six types of puzzles, similar to:

The Sliding Tile Puzzle: Imagine a 3x3 grid of numbers. You see the starting numbers, then they get covered up. The tiles slide around. You have to remember the starting numbers and track the slides to know what the final grid looks like.
The Cup Game: Like the "shell game" where a ball is hidden under one of three cups. The cups get swapped around. The AI has to track which cup has the ball.
The File Manager: Watching a screen where files are being created, deleted, and moved in a folder. The AI has to know exactly what files are left at the end.

The Catch: In these videos, the "secret state" (the numbers, the ball, the files) is often hidden for most of the video. The AI can only see the actions (the sliding, the swapping, the typing). It has to build a mental model of the invisible world based on the visible actions.

3. The Three Levels of Difficulty

The benchmark tests the AI on three levels of thinking, getting harder each time:

Level 1 (The Memory Test): "What was the very first move?" or "How many times did the red circle go up?" This is just recalling what you saw.
Level 2 (The Detective Work): "Based on all the moves you saw, what is the final arrangement of the numbers?" The AI has to infer the hidden state.
Level 3 (The Fortune Teller): "If we start from where the video ended, and we do these new moves, what will the result be?" This requires predicting the future based on a complex mental simulation.

4. The Results: The "Thinking" Gap

The researchers tested 18 of the smartest AI models available (including big names like GPT-4o and various open-source models).

The Struggle: Most models failed miserably. Many got less than 10% correct. Some got almost 0%. It was like asking a calculator to solve a physics problem without knowing the laws of physics.
The "Thinking" Breakthrough: They found that models with a special "Thinking Mode" (where the AI pauses to write out its reasoning steps before answering) performed much better.
- Gemini-2.5-Pro was the star of the show, scoring 56%.
- GPT-4o (without the special thinking mode) only scored 6.9%.

The Analogy: Imagine a student taking a math test.

Standard AI: Jumps straight to the answer. If the problem is hard, they guess.
Thinking AI: Writes out, "Okay, step 1 is X, step 2 is Y..." and checks their work. This "extended thinking" is the only thing that helped them solve these video puzzles.

5. Why This Matters

The paper proves two huge things:

AI needs to "think" to understand video: Just looking at a video isn't enough. To understand complex cause-and-effect, the AI needs to simulate the events in its mind, step-by-step.
Current AI is still "blind" to details: Even the best models struggle to track fine details over time. If you remove the video and just tell them the story in text, they get much better at the logic. This means the problem isn't the logic; it's the vision. They are bad at watching and remembering the visual details simultaneously.

The Bottom Line

VideoReasonBench is a new, very hard test that shows our current AI is still a bit clumsy when it comes to watching a video and figuring out a complex story. It's like a toddler who can recognize a dog but can't follow a game of hide-and-seek.

The good news? The models that are allowed to "think out loud" (like Gemini-2.5-Pro) are starting to get it. This benchmark will help developers build smarter AIs that can truly understand the world, not just recognize pictures.

1. Problem Statement

Despite recent advances in Long Chain-of-Thought (CoT) reasoning significantly boosting Large Language Models (LLMs) on complex tasks like mathematics and coding, this benefit has not been fully realized in video understanding. Existing video benchmarks (e.g., Video-MME, TempCompass, MMVU) suffer from two main limitations:

Lack of Reasoning Depth: They often rely on knowledge retrieval or short-horizon temporal understanding (e.g., counting actions, ordering events) rather than deep, multi-step reasoning.
Weak Visual Dependency: Many tasks can be solved using prior knowledge or textual summaries without strictly analyzing the visual content, failing to test the model's ability to track state changes over time.

Consequently, current benchmarks do not effectively evaluate whether Multimodal Large Language Models (MLLMs) can perform vision-centric complex reasoning, where the model must precisely track a sequence of operations on a latent state to infer or predict outcomes.

2. Methodology: VIDEOREASONBENCH

The authors introduce VIDEOREASONBENCH, a new benchmark designed specifically to test vision-centric, complex video reasoning.

A. Core Task Design

The benchmark conceptualizes videos as a sequence of state transitions: $\{S_t, o_t, S_{t+1}\}$ .

Latent State: The underlying state (e.g., a board configuration) is only partially visible (either at the start or the end of the video).
Observable Operations: The sequence of operations (e.g., sliding tiles, swapping cups) is fully visible.
Challenge: The model must track the operations to infer the hidden state or predict future states.

The benchmark features six video categories with varying visual complexities:

Number: Sliding tile puzzles on an $N \times N$ grid.
Circle: A red circle flips colors of grid pieces and neighbors.
Cup: Swapping cups to reveal hidden coins.
File: Simulated Linux command-line file operations (create, delete, move).
Card: Adding/removing cards from piles.
Chip: Adding/removing chips from cups.

B. Reasoning Levels

Questions are categorized into three escalating levels of difficulty:

Level 1 (Recall): Fine-grained visual perception. Requires recalling the exact sequence of operations or counting specific actions (e.g., "How many times was 'upward' moved?").
Level 2 (Infer): Latent state inference. Requires deducing the state at a specific time based on observed operations (e.g., "What is the board arrangement at the end?").
Level 3 (Predict): Counterfactual reasoning. Requires predicting a future state after hypothetical operations or finding the operation sequence to reach a target state.

C. Data Construction

Scale: 240 videos and 1,440 questions (6 categories $\times$ 3 levels $\times$ 2 skills per level).
Automation: A semi-automatic pipeline uses a Video Engine (generating state transitions via Matplotlib, CLI screenshots, or manual recording) and a Question Engine (generating questions and ground-truth answers based on the transition script).
Complexity Control: Difficulty is tunable via the number of operations (5–14) and state size ( $3\times3$ to $4\times4$ ).

3. Experimental Setup

Models Evaluated: 18 state-of-the-art MLLMs, including:
- Proprietary: GPT-4o, Gemini-2.0/2.5 (Flash & Pro), o4-mini, Seed1.5-VL.
- Open-source: LLaVA-OneVision, LLaVA-Video, InternVL3, Qwen2.5-VL, MiniCPM-V, mPLUG-Owl3, Kimi-VL.
Evaluation: A text-only LLM (Qwen2.5-72B) acts as a judge to verify answers against ground truth. For "Predict Operation" tasks, the model's output is simulated to check if it reaches the target state.
Baselines: Includes "vid2txt" (replacing video with text summaries) to isolate visual perception from reasoning capabilities.

4. Key Results

A. Performance Gap

General Failure: Most SOTA MLLMs struggle significantly. Open-source efficient models (<10B) score near random (<2%). Even flagship models like GPT-4o and Qwen2.5-VL-72B score below 10% (6.9% and 7.2% respectively).
Thinking Models: "Thinking-enhanced" models (o4-mini, Seed1.5-VL) show modest improvement (~10-11%) but still fail to master the task.
Top Performer: Gemini-2.5-Pro is the only model to demonstrate strong capability, achieving 56.0% accuracy. This highlights a significant gap between frontier proprietary models and current open-source systems.

B. Analysis of Failure Modes

Insufficient Temporal Perception: Models fail to track dense sequential operations. When videos are replaced with text summaries ("vid2txt"), performance jumps drastically (e.g., Seed1.5-VL from 11.5% to 69.4%), proving the bottleneck is visual tracking, not just reasoning logic.
Limited Multi-hop Reasoning: Non-thinking models fail to chain inferences. Enabling "Thinking" modes improves performance, but only if the model can first perceive the visual details correctly.

C. Test-Time Scaling & Vision Reliance

Thinking Budget: Increasing the "Thinking Budget" (token limit for reasoning) yields minimal gains (<2.5%) on existing benchmarks (Video-MME, TempCompass) but provides a ~9% gain on VIDEOREASONBENCH. This confirms that extended CoT is essential only for tasks requiring deep reasoning.
Vision Dependency: Removing visual input (cutting 50% of frames or using single frames) causes a catastrophic drop in performance on VIDEOREASONBENCH (e.g., 55.5% drop for 50% cut, 98.2% drop for single frame), whereas existing benchmarks show much smaller drops (<10%). This proves VIDEOREASONBENCH is strictly vision-centric.

5. Key Contributions

New Benchmark: Introduction of VIDEOREASONBENCH, the first benchmark explicitly designed to evaluate vision-centric complex reasoning with three progressive levels (Recall, Infer, Predict).
Empirical Findings: Revealed that current SOTA MLLMs (including thinking models) largely fail at complex video reasoning, with most scoring <10%. Only Gemini-2.5-Pro shows significant competence.
Insight on CoT: Demonstrated that extended Chain-of-Thought reasoning is ineffective on standard video benchmarks but is critical for complex video reasoning tasks.
Visual Necessity: Proved that complex video reasoning relies heavily on fine-grained visual perception, as removing visual context leads to near-total performance collapse.

6. Significance

This work challenges the community to move beyond "knowledge-driven" video QA. It establishes that true video reasoning requires models to simultaneously maintain a precise visual memory of state transitions and perform multi-step logical inference. The benchmark serves as a rigorous stress test for future MLLMs, indicating that current models lack the necessary fine-grained temporal perception and deep reasoning integration to handle dynamic, state-changing visual environments. The results suggest that future progress in video understanding will depend on improving both visual tracking mechanisms and the ability to utilize extended reasoning chains effectively.