From Perception to Action: An Interactive Benchmark for Vision Reasoning

This paper introduces CHAIN, an interactive 3D physics-driven benchmark that shifts Vision-Language Model evaluation from passive perception to active problem-solving, revealing that even top-performing models struggle to internalize physical constraints for reliable long-horizon planning and action execution.

The Gist

Imagine you are teaching a robot how to build a complex LEGO castle or solve a tricky wooden puzzle.

For a long time, we tested these robots by showing them a single, static photo of the finished castle and asking, "What is this?" or "How many blocks are there?" The robot would look at the picture, guess the answer, and get a score. This is like asking a chef to identify a cake just by looking at a photo of it, without ever letting them touch the ingredients or taste the batter.

The Problem:
Real life isn't a photo. Real life is messy, physical, and full of rules. If you try to put a big block on top of a tiny one, it falls. If you try to pull a piece out of a puzzle the wrong way, it gets stuck. The old tests didn't check if the robot understood these physical rules. They only checked if the robot could recognize things.

The Solution: CHAIN
The authors of this paper built a new test called CHAIN (Causal Hierarchy of Actions and Interactions). Think of CHAIN as a digital playground or a video game simulator where robots have to actually do things, not just talk about them.

Instead of a photo, the robot is dropped into a 3D world with physics (gravity, collisions, friction). It has to:

1. Look at the scene.
2. Think about what it can physically do next (e.g., "If I move this block, that one will fall").
3. Act (move, rotate, stack).
4. Watch what happens and adjust its plan if things go wrong.

The Two Main Games:
The benchmark uses two types of challenges to test the robots:

1. The Interlocking Puzzles (The "Lockpick" Test):
   Imagine a complex wooden puzzle where pieces are locked together like a 3D jigsaw. To take it apart, you can't just pull randomly. You have to find the one specific piece that unlocks the rest, then move it in a specific direction, which unlocks the next piece.

   • The Challenge: It requires understanding hidden rules. If you pull the wrong piece, the whole thing jams.
   • The Result: Even the smartest AI models (like GPT-5 or Claude) got stuck almost immediately. They couldn't figure out the "secret handshake" needed to unlock the puzzle.

2. The Stacking Game (The "Tetris" Test):
   Imagine you have a box and a pile of oddly shaped blocks. You have to pack them all in without them falling over or leaving gaps.

   • The Challenge: You can't just throw blocks in. You have to plan ahead. If you put a big block in the corner now, you might not have space for the last piece later.
   • The Result: The AIs were okay at easy levels but failed miserably at hard levels. They would greedily grab the "easy" blocks first, only to realize 10 steps later that they had trapped themselves with no way to finish the job.

The Big Surprise:
The researchers also tested "World Models" (AI that tries to predict what a video of an action would look like). They asked these AIs to generate a video of someone taking apart a puzzle.

• The Disaster: The AIs hallucinated wildly. They made blocks pass through each other like ghosts, merged two blocks into one, or made pieces disappear. They looked cool visually, but they were physically impossible. It was like watching a movie where the laws of physics were broken.

The Takeaway:
The paper concludes that current AI is great at seeing (recognizing a cat in a photo) but terrible at doing (understanding that if you push the cat, it will move).

• Old AI: "I see a puzzle. I know what a puzzle is."
• Real-World AI (What we need): "I see a puzzle. I know that if I pull piece A, piece B will slide left, but only if I rotate piece C first. If I don't, the whole thing breaks."

Why This Matters:
If we want robots to help us in the real world—fixing cars, building houses, or helping in hospitals—they need to pass the CHAIN test. They need to understand that the world has rules, and that every action changes the future. Right now, our robots are like toddlers who can name objects but don't understand how the world works yet. This benchmark is the first step to teaching them the difference.

Technical Summary

1. Problem Statement

Current evaluations for Vision-Language Models (VLMs) and diffusion-based world models predominantly rely on static, single-turn setups (e.g., Visual Question Answering or VQA). These benchmarks assess a model's ability to recognize objects or answer questions based on a single image but fail to evaluate interactive physical reasoning.

Real-world physical problem solving (e.g., embodied agents, robotic manipulation) requires:

• Dynamic Interaction: Agents must manipulate objects, observe feedback, and adapt plans.
• Constraint Awareness: Feasibility is determined by complex geometric structures, contact points, and support relations (e.g., gravity, interlocking).
• Long-Horizon Planning: Early actions must preserve the "feasible action space" for future steps.

Existing benchmarks do not test whether models can internalize these physical constraints to plan and execute multi-step action sequences in a physics-driven environment.

2. Methodology: The CHAIN Benchmark

The authors introduce CHAIN (Causal Hierarchy of Actions and Interactions), an interactive, 3D, physics-driven benchmark designed to shift evaluation from passive perception to active problem solving.

A. Task Families

CHAIN comprises two primary task families, totaling 109 distinct interactive levels:

1. Interlocking Mechanical Puzzles (32 instances):
   • Inspired by traditional puzzles like Kongming and Lu Ban locks.
   • Goal: Assemble or disassemble multi-piece structures via fine-grained manipulation (pick, rotate, insert).
   • Challenge: Requires reasoning about kinematic feasibility, collision avoidance, and hidden geometric constraints where the order of operations is critical.
   • Difficulty: Scaled from 6-piece locks (Easy) to complex 30+ piece designs (Hard).
2. 3D Stacking and Packing (77 instances):
   • Goal: Place diverse geometric blocks into a fixed container (e.g., a cube) to achieve exact volume coverage without overlap.
   • Challenge: Requires global spatial reasoning, stability analysis under gravity, and managing how early placement decisions restrict future free space.
   • Generation: Programmatically generated to ensure systematic control over complexity and scalability.

B. Construction Pipeline

• Physics Engine: Built using Unity (for complex interlocking mechanics) and a lightweight 3D Python engine (for stacking).
• Interaction Protocol: A closed-loop, multi-step interaction where the agent:
  1. Receives a task instruction and multi-view visual observations.
  2. Selects an action from a predefined set (color-coded object selection).
  3. Receives new observations based on the simulator's state update.
  4. Repeats until the goal is reached or a step budget (30–60 steps) is exceeded.
• Difficulty Calibration: Tasks are categorized into Easy, Medium, and Hard based on human expert solve times and structural complexity.

C. Evaluation Metrics

The benchmark evaluates models across three dimensions:

1. Task Success: Pass@1 (fraction of tasks solved in a single attempt).
2. Plan Efficiency: Measures the quality of the solution path for solved tasks, including AvgSteps, Distance-to-Optimal (excess steps), and Normalized Distance.
3. Cost Efficiency: Evaluates deployment trade-offs via Solved/Tokens and Solved/USD (monetary cost per success).

3. Key Contributions

1. Introduction of CHAIN: The first interactive 3D benchmark that evaluates VLMs and diffusion models on closed-loop physical reasoning, moving beyond static image understanding.
2. Unified Empirical Study: A comprehensive evaluation of state-of-the-art closed-source (e.g., GPT-5.2, Claude-Opus-4.5, Gemini-3-Pro) and open-source (e.g., Qwen3-VL, Kimi-k2.5) models, as well as video generation models (e.g., SORA, WAN, HUNYUANVIDEO).
3. Identification of Critical Gaps: The study reveals that current models struggle to internalize physical structure, often failing to translate perceived geometry into valid action sequences, particularly in long-horizon tasks.

4. Experimental Results

A. VLM Performance

• Overall Struggle: Even top-performing models (GPT-5.2) achieve low success rates on the full benchmark (Pass@1 ≈ 22.9%).
• Task Disparity: Models perform significantly better on Stacking (31.2% for GPT-5.2) than on Puzzles (3.1% for GPT-5.2). This indicates that interlocking mechanical reasoning is a major bottleneck.
• Efficiency vs. Success: Stronger models often exhibit "backtracking" behavior, increasing token usage and cost without guaranteeing success.
• Interactive vs. One-Shot: Interactive settings significantly outperform one-shot (static) settings. For Puzzles, one-shot accuracy drops to 0.0% for all models, proving that iterative constraint discovery is essential.

B. World Model (Video Generation) Failure

• Catastrophic Hallucinations: When tasked with generating step-by-step disassembly videos of Lu Ban locks, no evaluated video model (SORA, WAN, VEO, etc.) succeeded.
• Failure Modes:
  • Physics Violations: Objects pass through each other or move in physically impossible ways (e.g., direct translation through interlocks).
  • Representational Collapse: Objects lose identity, merge, or disappear as complexity increases.
  • Conclusion: Current world models lack the ability to reason about multi-step causal constraints and object permanence in structured 3D environments.

C. Impact of Difficulty

• Puzzles: Performance collapses immediately as difficulty increases; even "Easy" puzzles are rarely solved.
• Stacking: Performance degrades gradually but collapses on "Hard" levels (best result ~6.3%), highlighting the difficulty of long-horizon spatial planning.

5. Significance and Future Directions

• Paradigm Shift: CHAIN establishes a new standard for evaluating AI, moving from "seeing" (static recognition) to "acting" (dynamic, constraint-driven reasoning).
• Bottleneck Identification: The results highlight that current models lack physical intuition and the ability to simulate the consequences of actions over time. They fail to understand how early choices constrain future possibilities.
• Research Direction: The paper suggests that future progress requires:
  • Better integration of physics engines into model training.
  • Development of verifier-style signals rather than just reward models for long-horizon planning.
  • Improved object-centric reasoning to handle hidden geometric constraints.

The CHAIN benchmark and its baselines are open-sourced to catalyze the development of physically grounded, structure-aware interactive agents.