PerceptionComp: A Video Benchmark for Complex Perception-Centric Reasoning

The paper introduces PerceptionComp, a manually annotated benchmark comprising 1,114 complex questions across 279 diverse videos that require multi-step, long-horizon perceptual reasoning, revealing that current state-of-the-art models and even humans struggle significantly with tasks demanding compositional logic and repeated visual evidence.

The Gist

🎬 The Big Idea: "Don't Just Glance, Look Deeply"

Imagine you are watching a movie. Most video quizzes you've seen before are like pop quizzes after a 30-second clip. You see a dog run, and the question asks, "What color was the dog?" You can answer that instantly without thinking too hard.

But PerceptionComp is different. It's like a detective mystery where the clues are scattered across a 10-minute movie. To solve the puzzle, you can't just glance at the screen once. You have to:

1. Remember a red car from minute 2.
2. Notice a specific person wearing a blue hat at minute 5.
3. Realize that at minute 8, that person got into the red car.
4. Finally, answer: "What was the license plate on the car before the person got in?"

If you only watched the movie once and tried to remember everything, you'd likely fail. You need to rewind, fast-forward, and cross-reference different parts of the video to piece the story together.

🧩 What is PerceptionComp?

The researchers (from Tsinghua, UW, and NTU) built a new "test" for AI video models. They wanted to see if AI could do complex, perception-centric reasoning.

• The Problem: Current AI models are great at recognizing a cat or a car, but they are terrible at connecting the dots over time in a messy, real-world video. They tend to "hallucinate" (make things up) or get lost when the video gets complicated.
• The Solution: They created a dataset of 279 high-complexity videos (like city walks, sports, and game streams) and 1,114 tricky questions.
• The Rule: To answer a question, the AI must gather evidence from multiple, separate moments in the video. No single moment holds the whole answer.

🏗️ How They Built the Test (The Construction Site)

Think of building these questions like building a complex Lego set:

1. Selecting the Chaos: They didn't pick boring, empty videos. They used a robot (SAM2) to find videos with lots of moving parts—crowds, fast motion, and lots of objects. It's like choosing a busy city street instead of an empty parking lot.
2. The "And" & "Then" Logic: They designed questions that force the AI to do two things:
   • Conjunctive (The "And"): "Find a man who is wearing a red shirt AND blue pants AND is holding a green bag." (If he's missing one, he's not the guy).
   • Sequential (The "Then"): "Find the woman who dropped her keys. Then, find the man who picked them up. Then, tell me what color his shoes were." (You can't answer step 3 without doing steps 1 and 2 correctly).
3. Human Proofreading: Humans spent 10–20 minutes per question to make sure the answer was 100% correct and that you couldn't guess it without watching the video carefully.

🧠 The Results: Humans vs. AI

The researchers tested this on humans and the smartest AI models available (like GPT-o3, Gemini, and Qwen).

The Human Test:

• Single View: If humans watched the video once and couldn't rewind, they got it wrong 81% of the time (basically guessing).
• Unlimited Rewind: If humans could rewind and think as long as they wanted, they got 100%.
• Time: Humans took 5 to 10 times longer to answer these questions than they did on previous video tests. This proves the test is actually hard!

The AI Test:

• The Score: Even the best AI models (like Gemini-3-Flash) only got about 46% correct.
• The Gap: Open-source models were even lower, around 30-35%.
• The Takeaway: AI is still struggling to "watch" a video the way a human detective does. They often miss a small detail in minute 2, which causes them to fail the question in minute 10.

🔍 Why Do AI Models Fail? (The "Hallucination" Trap)

The paper found that AI models fail in two main ways:

1. The "One-View" Blindness: AI models often try to answer based on a single snapshot of the video. They miss the fact that the answer requires connecting two different times.
2. The "Over-Thinker" vs. The "Fast-Thinker":
   • Some AI models try to think too hard. They get confused by too many details and start making up stories (hallucinations) to fill in the gaps.
   • Interestingly, sometimes the "lighter" models (like Gemini-Flash) did better than the "heavy" ones (Gemini-Pro) because the heavy ones got bogged down in unnecessary details and lost the main thread.

🚀 What Does This Mean for the Future?

The paper suggests that to make AI truly understand videos, we can't just make them "smarter" with more brain power. We need to teach them to:

• Pause and Rewind: Treat video like a book you can flip back and forth in, not a stream you just watch once.
• Connect the Dots: Learn to hold a piece of information in their "memory" from minute 1 and use it to solve a problem in minute 5.

In a nutshell: PerceptionComp is a new, very difficult gym for AI video models. It shows us that while AI is getting good at seeing, it's still terrible at remembering and connecting what it sees over time. We have a long way to go before AI can be a true "video detective."

Technical Summary

1. Problem Statement

Current video understanding benchmarks fail to accurately evaluate long-horizon, perception-centric reasoning. Existing benchmarks fall into two categories:

• Perception-centric but easy: Models (and humans) can often solve these by viewing the video once or relying on language priors (e.g., VideoMME, Perception Test).
• Logic-heavy but visually simple: These require complex reasoning but use synthetic or overly simplified visuals where the bottleneck is logic, not visual perception (e.g., maze solving, geometry).

There is a critical gap: no existing benchmark forces models to perform repeated visual information gathering across temporally separated segments to solve a query. Deep video understanding requires not just "thinking" (reasoning) but "re-watching" (perception) to integrate evidence distributed over time.

2. Methodology: The PerceptionComp Benchmark

The authors introduce PerceptionComp, a fully manually annotated benchmark designed to force models to repeatedly revisit a video to gather evidence.

A. Data Construction

• Video Selection: 279 high-complexity videos were selected from diverse domains (city walks, mall tours, sports, home tours, movies, games).
  • Complexity Metrics: Videos were filtered using automatic proxies: SAM2 instance counts (object density) and optical-flow magnitude (motion intensity). This ensures scenes are cluttered, dynamic, and difficult to summarize with a single caption.
  • Duration: 2 to 10 minutes.
• Question Design: 1,114 five-choice multiple-choice questions were created.
  • Compositional Logic: Questions combine multiple subconditions using Conjunctive (all must be true) or Sequential (order-dependent) logic.
  • Perceptual Skills: Subconditions target specific skills: semantic recognition, visual correspondence (tracking), temporal reasoning, spatial reasoning, visual knowledge, and world modeling.
  • Constraint: No single moment is sufficient. Solving a question requires integrating evidence from multiple, temporally separated segments.
• Annotation Protocol:
  • Fully Manual: Each question took 10–20 minutes to annotate.
  • Verification: Annotators ensured the answer was unique and that every subcondition was necessary (no proper subset of conditions could solve the query).
  • Difficulty Levels: Questions are categorized into three difficulty levels based on the number and complexity of subconditions.

B. Evaluation Setup

• Human Baselines:
  • Expert (Unrestricted Rewatch): 100% accuracy.
  • Human (Single-View): Accuracy drops to 18.97% (near random chance), proving the task cannot be solved by single-pass memory or language priors.
• Model Evaluation: Evaluated on a wide range of State-of-the-Art (SOTA) Multimodal Large Language Models (MLLMs), including proprietary models (Gemini-3, GPT-o3, GPT-4o) and open-source models (Qwen-VL, InternVL, Video-R1).

3. Key Contributions

1. New Benchmark Paradigm: Shifts the focus from "memory" or "pure logic" to perception-centric reasoning, where the difficulty arises from the need to repeatedly gather and integrate fine-grained visual evidence.
2. High-Quality Dataset: A manually curated dataset of 1,114 complex questions over 279 high-dynamic videos, ensuring that shortcuts (like single-view memorization) are ineffective.
3. Comprehensive Analysis: Provides a deep dive into model failure modes, specifically analyzing the impact of perception budgets (number of input frames) and reasoning budgets (thinking tokens).

4. Key Results

• Model Performance:
  • Even the best-performing model (Gemini-3-Flash) achieved only 45.96% accuracy.
  • Open-source models generally scored below 40%.
  • Scaling Issues: Increasing model size did not consistently improve performance, suggesting the bottleneck is not capacity but the ability to extract fine-grained evidence under clutter.
• Impact of Test-Time Compute:
  • Reasoning: "Thinking" models (e.g., GPT-o3) outperformed their non-thinking counterparts (GPT-4o by ~11%), but still fell far short of human levels.
  • Perception: Increasing the number of input frames (e.g., from 16 to 64) significantly improved accuracy for both GPT-o3 and Qwen3-VL, confirming the benchmark's reliance on dense temporal evidence.
• Human vs. AI Gap:
  • Humans achieve 100% with rewatching but drop to ~19% without it.
  • Top AI models remain in the mid-40s, indicating a fundamental limitation in long-horizon, perception-driven reasoning.

5. Error Analysis & Failure Modes

The paper identifies specific bottlenecks in current frontier models:

• Mid-Chain Collapse: Errors often occur in the middle of the reasoning chain (Steps 2–3).
• Spatial Misunderstanding: A significant portion of failures (60% of mid-stage errors) stems from incorrect spatial reasoning (e.g., wrong relative positions, 3D relationships).
• Error Propagation: Once a model misidentifies an entity or state in an early step, the subsequent reasoning chain becomes internally coherent but factually wrong (hallucination).
• Pro vs. Flash Anomaly: Interestingly, Gemini-3-Flash outperformed Gemini-3-Pro. The authors attribute this to the "Streamlining Effect": the Pro model's tendency to over-analyze fine-grained details led to logical hallucinations, while Flash's focused, direct inference avoided unnecessary noise.

6. Significance

• Diagnostic Tool: PerceptionComp serves as a focused testbed to diagnose the specific limitations of MLLMs in visual thinking, distinguishing between models that can "talk" about videos and those that can truly "see" and "reason" about them over time.
• Future Direction: The results suggest that improving video understanding requires not just better language reasoning, but better perception grounding and the ability to sustain iterative evidence gathering.
• Community Impact: It highlights that current SOTA models are still far from human-level performance in complex, real-world video scenarios, guiding future research toward robust spatio-temporal modeling and iterative perception strategies.