🤖 AI

IMBench: A Benchmark for Intuitive Robotic Manipulation

This paper introduces IMBench, a comprehensive benchmark comprising 35 tasks and 14K trajectories designed to evaluate the integrated capability of "intuitive manipulation"—combining physical reasoning, perception, and action generation—revealing significant gaps in current vision-language and vision-language-action models' ability to plan and execute complex, constraint-rich robotic tasks.

Original authors: Anurag Maurya, Sukhvansh Jain, Prajwal Avhad, Gautham Balachandran, Ziyi Zhou, Atharva Kshirsagar, Satyam Singh, Bowen Li. Rishabh Mukund, Ritul Singh, Jatin Vira, Suvonil Chatterjee, Devesh K. Jha

Published 2026-07-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anurag Maurya, Sukhvansh Jain, Prajwal Avhad, Gautham Balachandran, Ziyi Zhou, Atharva Kshirsagar, Satyam Singh, Bowen Li. Rishabh Mukund, Ritul Singh, Jatin Vira, Suvonil Chatterjee, Devesh K. Jha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots are like incredibly talented actors who have memorized millions of scripts. They can recite lines perfectly and move their arms with precision if the stage is set exactly as they've seen before. But ask them to improvise a scene where the floor is slippery, or a prop is heavier than it looks, and they often freeze or stumble. This is the current state of robotics: they are great at following instructions but struggle with "intuition."

To understand why this matters, we need to look at two superpowers humans have that robots currently lack. The first is intuitive physics. This isn't a complex math class; it's the brain's ability to guess how things will move, fall, or bounce just by looking at them. It's knowing that a tall, wobbly stack of books will tip over if you push it too hard, or that a thin plate on a table can't be grabbed from the top without sliding it to the edge first. The second is means-end reasoning, which is the ability to figure out how to get from where you are to where you want to be, even if the direct path is blocked. It's the difference between trying to grab a cookie from a high shelf and realizing you need a chair first.

Scientists care about this because for robots to truly help us in messy, real-world homes, they can't just be pre-programmed machines. They need to understand the physical world the way we do, so they can adapt when things don't go according to plan. If a robot can't figure out that a cup is too slippery to hold, or that a door is stuck, it's not very useful.

Enter IMBench, a new "test drive" for robots designed by researchers at Manav Robotics. Think of IMBench not as a standard exam with multiple-choice questions, but as a series of tricky, real-life puzzles. The researchers built 35 different scenarios—like balancing a rod on a tiny ridge, using a hook to pull a toy out of reach, or stacking weirdly shaped blocks without them toppling over. These tasks are designed to trick robots that rely only on memorizing patterns. They force the robot to pause, think about the physics, and come up with a new plan on the fly.

The paper tests three different types of "brains" on these puzzles. First, they asked powerful AI chatbots (Vision-Language Models) to just look at the scene and explain what's happening. Surprisingly, these AI models were pretty good at spotting the rules. They could tell you, "Hey, that plate is too thin to grab from the top," about 74% of the time. They understood the theory.

However, the plot thickens when the researchers asked these same AIs to make a plan to solve the problem. The success rate dropped. The AI could explain the problem but struggled to figure out the exact steps to fix it. It was like a student who could explain the rules of soccer perfectly but couldn't figure out how to kick the ball into the goal.

Finally, the researchers tested actual robot control systems (policies) that were supposed to execute the actions. Here, the results were quite humbling. Even the most advanced robot policies, which had been trained on thousands of examples, failed miserably on these intuitive tasks. When asked to balance a rod or use a tool, they succeeded less than 25% of the time, and often not at all. The paper suggests that while these robots are getting better at copying human movements, they are still missing that crucial spark of physical intuition. They can mimic the action, but they don't truly understand why the action works.

The researchers conclude that we are missing a key piece of the puzzle. Current robots are like actors who can memorize a script but can't improvise. IMBench shows us exactly where they are failing: in the gap between understanding a physical problem and actually solving it with their hands. It's a reminder that for robots to become our true partners, they need to learn not just how to move, but how to think about the world they are moving in.

Technical Summary: IMBench – A Benchmark for Intuitive Robotic Manipulation

Problem Statement

Current robotic learning systems have advanced toward "generalist" robots capable of diverse tasks, yet they lack the ability to ground behavior in the physical world as humans do. Humans integrate intuitive physics (approximate mental simulation of object dynamics, forces, and contacts) with means-end reasoning (planning based on goals and constraints) to convert physical understanding into goal-directed action. This capability, termed intuitive manipulation, allows humans to adapt to new scenes, infer hidden constraints, and execute complex, contact-rich sequences.

Existing benchmarks fail to evaluate this capability holistically. Physical reasoning suites (e.g., PhysBench, CLEVRER) evaluate inference in isolation without requiring executable action. Conversely, manipulation benchmarks (e.g., LIBERO, RoboCasa) focus on policy execution and generalization but do not isolate whether success depends on reasoning about specific physical constraints. Consequently, there is a missing axis in current foundation models and generalist robot policies: the integration of physical reasoning with low-level execution.

Methodology: IMBench

The authors introduce IMBench, a benchmark designed to evaluate intuitive manipulation as an integrated capability spanning perception, physical reasoning, action generation, and iterative execution.

Task Suite

IMBench consists of 35 manipulation tasks built on the RoboSuite simulator. The tasks are organized around an Understand–Infer–Act cognitive loop:

Understand: Extract latent physical properties (mass, friction, geometry, contact) from perception.
Infer: Simulate forward dynamics to anticipate failures and construct sub-goals.
Act: Execute closed-loop, contact-rich policies with proprioceptive and force feedback.

The 35 tasks are categorized into seven distinct inference bottlenecks:

P1 (Geometry): Constrained grasping (e.g., edge-sliding thin plates, suction-to-jaw handovers).
P2 (Dynamics): Trajectory prediction (e.g., tossing, pendulum grasping, bouncing interception).
P3 (Causal): Indirect action (e.g., domino cascades, seesaw balancing, transitive pushing).
P4 (Tool Use): Augmented reach (e.g., hook-dragging, lever use).
P5 (Hidden State): Active discovery (e.g., probing mass, uncovering occluded objects).
P6 (Reactive): Replanning under perturbation (e.g., slip recovery, stack collapse).
P7 (Stability): Equilibrium construction (e.g., balancing rods, zero-gap packing).
Miscellaneous: Tasks testing frame decoupling (mirror-pick-place) and bimanual coordination (keyboard typing).

Dataset and Infrastructure

Data: The benchmark includes ~14,000 filtered trajectories (6,000 teleoperated human demonstrations and 8,000 scripted oracle trajectories).
Curation: A three-stage filtering pipeline ensures quality: human annotation (1–5 scale, keeping ≥4), VLM-based anomaly detection, and final human verification.
Modalities: Data includes synchronized multi-view RGB, proprioception, gripper states, and wrist force/torque signals.
Evaluation Protocol: The benchmark evaluates models across three stages:
1. Constraint Understanding: Can a Vision-Language Model (VLM) identify physical constraints?
2. Plan Proposal: Can the model generate a verifiable sequence of sub-goals?
3. Execution: Can a visuomotor policy execute the plan in a closed-loop setting?

Key Contributions

IMBench Benchmark: A set of 35 tasks specifically designed to test whether models can translate physical understanding into action, highlighting where current foundation models and learned policies fail.
Dataset and Tools: Release of ~14K curated trajectories and scripts for generating diverse scenarios to support reproducible research.
Formalization and Evaluation: A formal decomposition of intuitive manipulation (Understand–Infer–Act) and a three-stage evaluation protocol that systematically measures the gap between understanding and execution.

Experimental Results

The authors evaluated frontier Vision-Language Models (VLMs) and state-of-the-art Vision-Language-Action (VLA) policies.

High-Level Reasoning (VLMs)

Constraint Understanding (Stage 1): Frontier VLMs (e.g., GPT-5.5, Claude Sonnet 4.6) show moderate to high accuracy (approx. 74% mean) in identifying physical constraints. They perform well on static geometry and causal chains but struggle with stability (P7) and specific grasp mechanics (e.g., cracker-box tasks).
Plan Proposal (Stage 2): Performance drops significantly when translating understanding into plans. While some geometric tasks transfer well, tasks requiring ordered or timed actions (e.g., slide-catch, tool-retrieve) see a sharp decline. Models often recognize a constraint but fail to sequence the necessary steps correctly.
Execution Gap: Even with correct high-level plans, the success rate for generating executable plans drops to ~11% when evaluated with a closed-loop agent (IMBAgent).

Low-Level Execution (Visuomotor Policies)

Zero-Shot Performance: State-of-the-art VLA policies (e.g., $\pi_0.5$ , GR00T) achieve near-zero success rates (<0.02 mean) in zero-shot settings.
Fine-Tuned Performance: Even after fine-tuning on IMBench data, policies struggle.
- Success: Policies perform best on contact-tolerant tasks (e.g., domino-single, mirror-pick-place for Diffusion Policy) and tasks with broad success regions.
- Failure: Policies fail on tasks requiring precise alignment, grasp mechanics, reorientation, or balancing (e.g., cup-inversion, shape-stack, balance-hard).
Out-of-Distribution (OOD) Generalization: Performance degrades sharply when physical factors (e.g., center of mass, object orientation) are perturbed. For instance, a policy achieving 71% on balance-medium drops to 12% under small center-of-mass shifts, suggesting reliance on memorized patterns rather than robust physical understanding.

Significance and Claims

The paper positions intuitive manipulation as a missing capability axis in current foundation models, orthogonal to the dimensions (semantic generalization, task completion) that current benchmarks reward.

The "Understanding-to-Action" Gap: The results reveal a consistent gap where models can describe physical constraints but cannot convert that understanding into executable, physically grounded plans.
Limitations of Current Approaches: Current VLAs and learned policies struggle to satisfy task constraints and generalize across scenarios when the task structure itself requires deep physical reasoning.
Role of IMBench: IMBench serves as a measurement instrument to help the field localize where intuitive physical intelligence is missing in current systems, moving beyond simple imitation or semantic reasoning toward adaptive physical intelligence.

The authors conclude that while current models show partial physical reasoning, the integration of this reasoning into robust, adaptive action remains an open challenge, necessitating benchmarks like IMBench that explicitly test the full pipeline from perception to closed-loop execution.