Spatial Causal Prediction in Video

This paper introduces Spatial Causal Prediction (SCP), a new task paradigm and benchmark (SCP-Bench) designed to evaluate and improve video models' ability to infer unseen spatial states and causal outcomes beyond visible observations, revealing significant gaps between current AI and human intelligence in this domain.

The Gist

Imagine you are watching a cooking show. The chef is mixing ingredients in a bowl. Suddenly, the video cuts to black right before the chef pours the mixture onto a plate.

The Question: "Based on what you just saw, where will the food land on the plate?"

Most current AI models are like a student who has memorized the recipe but has never actually cooked. If you ask them, "What is in the bowl?" they can tell you. But if you ask, "Where will the food go after the cut?" they often guess wrong because they don't truly understand physics or cause and effect. They just see the picture, not the story.

This paper introduces a new way to test AI called SCP (Spatial Causal Prediction). Here is the breakdown in simple terms:

1. The Problem: The "Static" vs. "Dynamic" Gap

Think of existing AI benchmarks as a photo album. They ask questions about things you can see right now: "How many bowls are there?" or "Is the knife on the left?"

• The Limitation: Real life isn't a photo album; it's a movie. Things move, collide, and change.
• The New Challenge: SCP asks the AI to be a fortune teller or a time traveler. It forces the AI to watch a video, stop it at a specific moment, and predict what happens next (or what happened before) based on the laws of physics and logic, not just by looking at pixels.

2. The Solution: SCP-Bench (The "Gym" for AI)

The researchers built a massive gym called SCP-Bench to train and test these AI models.

• The Equipment: They collected 1,181 video clips from sports, driving, factories, and kitchens.
• The Workout: They created 2,500 questions. Some ask the AI to predict the future (e.g., "Will the ball go left or right?"), and some ask it to reconstruct the past (e.g., "Which object did the person touch first?").
• The Twist: The AI is only allowed to see part of the video. It has to fill in the missing pieces using logic.

3. The Results: The "Reality Check"

The researchers tested 23 of the smartest AI models available (including big names like GPT-5, Gemini, and Qwen). Here is what they found:

• The Gap is Huge: Humans scored about 90% on these tests. The best AI model only scored about 66%. That's a massive gap. It's like a human chess grandmaster playing against a very good, but still amateur, computer.
• Size Matters (But isn't everything): Bigger models generally did better, like a student with a bigger library of books. However, even the biggest models struggled with the "physics" part.
• The "Thinking" Trap: The researchers tried forcing the AI to "think step-by-step" (like a human solving a math problem). Surprisingly, this didn't help much. It's like telling a confused student to "write down their thoughts"—if they don't understand the concept, writing it down doesn't fix the error.
• Perception vs. Reasoning: The study found that the AI isn't failing because it can't see the video (perception). It fails because it can't reason about what it sees. It sees a ball moving up, but it doesn't "know" gravity will pull it down.

4. How to Fix It? (The "Training Wheels")

The researchers tried a few tricks to help the AI:

• Giving Hints: When they gave the AI a text description of the future (e.g., "The ball will fall down"), the AI got much better. This suggests the AI is smart enough to use the answer if it's given the right clues, but it can't generate those clues on its own yet.
• More Data: Simply making the models bigger helped, but it wasn't a magic bullet.

The Big Takeaway

This paper is a wake-up call. We have built AI that is amazing at describing what it sees (like a tour guide), but it is still terrible at understanding how the world actually works (like a physicist).

The Analogy:

• Current AI: A very observant tourist who can describe the scenery perfectly but doesn't understand why the river flows downstream.
• Human Intelligence: Someone who understands the river flows downstream because of gravity, and can predict where a leaf will end up even if they can't see it yet.

The authors are saying: "We need to stop just teaching AI to see and start teaching it to understand cause and effect." Until we do that, our self-driving cars and robots might still trip over their own feet when things get complicated.

Technical Summary

1. Problem Definition

Current Multimodal Large Language Models (MLLMs) excel at understanding visible spatio-temporal attributes (e.g., object location, size, or relations within a static frame or a fully observable video). However, they struggle with Spatial Causal Prediction (SCP): the ability to infer unseen past states or predict future spatial outcomes based on partial observations and causal dynamics.

Real-world applications like autonomous driving and robotics require systems to reason beyond what is immediately visible, understanding physical laws, causality, and dynamic evolution. Existing benchmarks primarily focus on "Seen" scenarios (static or fully observable dynamic scenes), failing to test a model's capacity for "Unseen" reasoning (inferring causes from effects or predicting effects from causes).

2. Methodology

A. SCP-Bench Construction

The authors introduce SCP-Bench, a high-quality benchmark designed to evaluate spatial causal intelligence.

• Scale: 2,500 Question-Answer (QA) pairs across 1,181 video clips.
• Data Sources: Aggregated from diverse datasets including Ego-Exo4D, HD-EPIC, YouTube-8M, and ActivityNet.
• Construction Pipeline:
  1. Selection: Automated screening using GPT-5 to identify clips with clear spatial changes and causal dynamics.
  2. Generation: Structured prompts generate QA candidates focusing on 8 spatial reasoning categories.
  3. Cutpoint Definition: A critical step where a "cutpoint" is defined to split the video into a visible portion (input) and an unseen portion (target for inference).
  4. Validation: Rigorous human annotation ensures questions are unambiguous, grounded in physical causality, and answerable only through reasoning, not direct observation.
• Task Dimensions:
  • 8 Question Types: Relative Speed, Counting, Planning, Relative Distance, Spatial State, Appearance Order, Relative Size, and Relation.
  • Causal Directions: Backward (inferring past causes from current state) and Forward (predicting future states from current observation).
  • Perspectives: Single-view (Ego/Exo) and Multi-view (Ego-Exo, Exo-Ego, etc.).
  • Scenes: Sports, Driving, Factory/Machine, Life Records, Artistic Performances, and Animal-related.

B. Experimental Setup

• Models Evaluated: 23 state-of-the-art MLLMs, including proprietary models (GPT-5, Gemini-2.5, Claude-4.5) and open-source models (Qwen3-VL, InternVL3.5, LLaVA variants, SpaceR).
• Evaluation Metrics: Accuracy on multiple-choice questions.
• Analysis Strategies:
  • Decomposition: Separating perception (visual input) from reasoning (logical inference).
  • Temporal Extrapolation: Testing performance across short, mid, and long time horizons.
  • Causal Consistency: Checking if models adhere to physical laws (e.g., pendulum motion).
  • Enhancement Strategies: Testing Model Scaling, Chain-of-Thought (CoT), Self-Think, Perception Enhancement (dense captions, spatial graphs), and Causal Scaffolds (text/image/video predictions of the future).

3. Key Contributions

1. New Task Paradigm (SCP): Formulated the first task specifically targeting the inference of unseen spatial states (past and future) based on causal dynamics, moving beyond standard "Seen" video understanding.
2. SCP-Bench: Created a rigorous, multi-dimensional benchmark with 2,500 QA pairs covering diverse viewpoints, scenes, and causal directions, filling a gap in existing spatial reasoning evaluations.
3. Comprehensive Analysis: Provided an in-depth evaluation of 23 models, revealing that current MLLMs are significantly weaker than humans in causal prediction and that reasoning, not just perception, is the primary bottleneck.
4. Strategic Insights: Identified that while standard CoT and self-thinking offer limited gains, scaling model size and providing unseen spatial causal scaffolds (especially textual descriptions of future states) significantly improve performance.

4. Key Results

Performance Gaps

• Human vs. Model: Humans achieve 89.61% accuracy, while the best model (GPT-5) reaches only 66.24%. There is a substantial gap (~23%) in unseen spatio-temporal causal prediction.
• Open vs. Closed: Large open-source models (e.g., Qwen3-VL-235B, InternVL3.5-241B) perform comparably to or better than some proprietary closed-source models on specific tasks.
• Task Difficulty: "Relative Size" is the easiest task; "Planning," "Relation," and "Counting" are the most difficult, requiring higher-order interaction reasoning.

Critical Findings

• Reasoning > Perception: When models are given the "Gold Video" (the unseen answer directly), performance improves significantly. However, when forced to rely on dense captions alone (removing visual input), performance drops but remains non-trivial. This indicates that spatial causal reasoning is the primary bottleneck, not just visual perception.
• Temporal Extrapolation: Models show little sensitivity to the length of the prediction horizon (short vs. long), suggesting they do not effectively learn temporal continuity.
• Causal Consistency: Models often fail to adhere to basic physical laws (e.g., a swing reversing direction at its peak). They frequently hallucinate trajectories that violate causality.
• Single-Frame Bias: Surprisingly, some models perform slightly better with a single cutpoint frame than with full video sequences, suggesting they rely on static cues rather than temporal dynamics.

Effectiveness of Improvement Strategies

• Model Scaling: Performance correlates positively with parameter count, though gains are non-monotonic for smaller models. Significant jumps occur at large scales (e.g., 30B+).
• Scaffolds: Providing textual descriptions of the future (generated by GPT-5) yields the largest performance boost (+13-16%), as MLLMs are better at reasoning over text than raw pixels. Video and image scaffolds offer smaller gains.
• CoT & Self-Think: Standard Chain-of-Thought and Self-Think mechanisms provide marginal or inconsistent improvements, sometimes even degrading performance due to noise.

5. Significance

This work fundamentally shifts the evaluation of spatial intelligence in AI. It demonstrates that current MLLMs are largely "perceptual" rather than "causal" agents. By introducing SCP-Bench, the authors provide a critical testbed for developing systems capable of embodied intelligence, which is essential for robotics and autonomous driving where predicting unseen outcomes is a safety-critical requirement. The findings suggest that future progress in spatial reasoning will depend less on better vision encoders and more on scaling reasoning capabilities and integrating physical commonsense and causal scaffolds into the inference process.