💬 NLP

ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop

This paper introduces ESI-Bench, a comprehensive benchmark for embodied spatial intelligence built on OmniGibson and Spelke's core knowledge systems, which demonstrates that active perception-action loops significantly outperform passive observation by enabling agents to uncover hidden spatial information, while revealing that current models fail primarily due to action blindness and metacognitive gaps rather than weak perception.

Original authors: Yining Hong, Jiageng Liu, Han Yin, Manling Li, Leonidas Guibas, Li Fei-Fei, Jiajun Wu, Yejin Choi

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yining Hong, Jiageng Liu, Han Yin, Manling Li, Leonidas Guibas, Li Fei-Fei, Jiajun Wu, Yejin Choi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery in a house where you can only see what's directly in front of your eyes. You can't walk around, you can't touch anything, and you can't peek behind the sofa. You have to guess what's in the kitchen, how many apples are on the counter, or if a ball is hidden inside a box just by looking at a single, frozen photo.

This is how most current AI "spatial intelligence" works. It's like a detective who is handcuffed to a chair, staring at one photo, trying to solve a crime.

Enter ESI-BENCH.

The authors of this paper built a new testing ground called ESI-BENCH (Embodied Spatial Intelligence Benchmark). Think of it as a giant, virtual playground where AI agents are given a body, legs, and hands. Instead of just staring at a photo, they are allowed to walk around, look up and down, pick things up, and move objects to answer questions.

Here is the breakdown of what they found, using simple analogies:

1. The "Active Detective" vs. The "Passive Observer"

The researchers tested AI models in three ways:

The Passive Observer: The AI gets one photo and has to guess.
The Passive Hoarder: The AI gets 30 random photos of the room (like someone flipping through a photo album randomly) and has to guess.
The Active Detective: The AI can choose where to walk, what to look at, and what to touch to solve the puzzle.

The Big Surprise:
The "Passive Hoarder" (getting 30 random photos) often did worse than the single photo. It was like giving a detective a stack of 30 blurry, irrelevant photos; it just confused them.
However, the Active Detective was a game-changer. Without being told how to solve the puzzle, the AI spontaneously figured out smart strategies.

Example: If asked "Is there a chestnut inside this glass?", the AI didn't just stare. It figured out it needed to walk behind the glass, look from the top, or even pick up the glass and pour it out to see. It invented these strategies on its own.

2. The Real Problem: "Action Blindness"

The paper discovered that the AI's eyes (perception) are actually pretty good. The real problem is its brain's ability to choose actions.

The Analogy: Imagine a person who has perfect vision but doesn't know where to look. They might stand in the corner and stare at a blank wall for 30 seconds, missing the treasure chest right behind them.
The Finding: When the researchers gave the AI the perfect path to walk (a "ground truth" trajectory), the AI solved the puzzles almost perfectly. This proves the AI could see the answer if it just knew where to go. The failure wasn't that it couldn't see; it was that it didn't know what to do next.

3. The "3D Glasses" Trap

The researchers tried giving the AI "3D glasses" (reconstructing the room in 3D from the photos) to help it understand depth.

The Result: When the 3D glasses were perfect, the AI got smarter. But when the 3D reconstruction was slightly noisy or imperfect (which happens often), the AI got dumber than before.
The Analogy: It's like giving someone a pair of glasses that distort the world slightly. Instead of helping them see better, the distortion makes them trip over things they could have easily avoided with just their normal eyes. The AI trusted the "broken" 3D map too much and made bad decisions.

4. The "Overconfident Guess" (The Human Gap)

The most interesting finding was about confidence.

Humans: When a human detective isn't sure, they say, "I need to look from another angle," or "Let me check behind that curtain." They are willing to change their mind if they find new evidence.
AI: The AI models are like a stubborn detective who guesses "It's a piano!" after one quick glance. Even when they walk around and see evidence that it's actually a cupboard, they refuse to change their mind. They double down on their first guess with 100% confidence, even when they are wrong.
The Metaphor: Humans are like explorers who keep their map flexible. The AI is like a tourist who looks at a sign once, decides the museum is closed, and refuses to walk inside to check, even if the door is wide open.

Summary of the Benchmark

The paper created a test with 10 categories of challenges, based on how human babies learn about the world (like understanding objects, numbers, and physics).

Counting: How many balls are hidden under a blanket?
Physics: Will this stack of blocks fall over?
Reflection: Is that object in the mirror real, or just a reflection?
Navigation: Can I get from the bedroom to the kitchen without going through the living room?

The Bottom Line

The paper concludes that to build truly smart robots, we can't just make their cameras better. We have to teach them how to move and interact. They need to learn that sometimes, to see the truth, you have to stop staring and start walking, touching, and exploring. Currently, AI is great at seeing what's in front of it, but terrible at knowing what it needs to do next to find the answer.

Technical Summary: ESI-BENCH

Problem Statement

Spatial intelligence is fundamentally an embodied process defined by a perception–action loop, where agents must actively acquire observations to reason about latent physical properties (e.g., occluded structures, containment, dynamics) that are inaccessible through passive sensing alone. Existing benchmarks for spatial reasoning largely assume "oracle" observations—fixed, pre-given inputs—failing to evaluate whether agents can determine which observations to acquire, when to stop, and how to sequence actions to resolve spatial ambiguities. Current models often suffer from "action blindness," where poor action selection leads to uninformative views, cascading into reasoning errors, and exhibit a metacognitive gap where they commit to answers prematurely with high confidence, unlike humans who seek falsifying evidence.

Methodology

The authors introduce ESI-BENCH, a comprehensive benchmark for embodied spatial intelligence built upon the OmniGibson simulator and the BEHAVIOR-1K scene pool. The benchmark is grounded in Spelke's four core knowledge systems (object representation, layout/geometry, number representation, and agents/goal-directed actions).

Benchmark Construction

Scale: 3,081 task instances across 10 task categories and 29 subcategories.
Task Definition: Each task is a tuple $(S, p_0, q, y^*)$ , where $S$ is a 3D scene, $p_0$ is the initial agent pose, $q$ is a natural language question, and $y^*$ is the ground-truth answer.
Action Space: Agents operate within a discrete high-level action space spanning Locomotion (moving/rotating), Perception (viewpoint rotation), and Manipulation (picking, placing, pouring, filling).
Generation Pipeline: GPT-4o is used as a proposal engine to select objects, determine placements, generate questions, and propose action trajectories. These proposals undergo rigorous simulator-based filtering (physics checks, collision detection) and human verification (correctness, answerability, non-triviality) to ensure physical validity and that tasks cannot be solved via visual bias or priors alone.
Evaluation Paradigms: The benchmark evaluates models under four distinct conditions to isolate sources of error:
1. Passive Single-View: Fixed initial observation (baseline).
2. Passive Multi-View: 30 randomly sampled views (simulating exhaustive but unselected observation).
3. Active Exploration: Agents choose actions to gather evidence within a 30-step budget.
4. Ground-Truth Passive: Agents are provided with the exact sequence of views from the optimal ground-truth trajectory (oracle ablation).

Experimental Setup

The authors evaluated state-of-the-art Multimodal Large Language Models (MLLMs), including GPT-5 and Gemini 3.1, in both 2D-only and 3D-augmented configurations (using VGGT for scene graph reconstruction). Human performance was also collected to establish an upper bound. All evaluations were conducted zero-shot.

Key Results

1. Active Exploration vs. Passive Observation

Emergent Strategies: Without explicit instruction, active agents spontaneously discovered diverse spatial strategies (e.g., moving behind objects, top-down views, manipulating containers) to resolve ambiguities.
Performance Gains: Active exploration substantially outperformed passive counterparts. For example, on View Hallucination, accuracy rose from 39.9% (passive) to 68.1% (active).
Noise vs. Signal: Passive multi-view (randomly sampling 30 views) often degraded performance or provided negligible gains compared to single-view baselines, suggesting that unstructured observation accumulation adds noise rather than signal.

2. Action Blindness Dominates Perceptual Blindness

The Bottleneck: For most tasks, the primary failure mode is action selection, not perception. When provided with ground-truth optimal views (oracle), model performance jumped significantly (e.g., Rigid Containment improved from 42.5% to 95.0% for GPT-5).
Cascading Errors: Suboptimal actions lead to uninformative views, which trigger worse subsequent actions, creating a compounding chain of errors that cannot be recovered within the step budget.
Hard Perceptual Ceilings: Certain tasks (e.g., Geometric Configuration, Specular Reflection) remain difficult even with oracle views, indicating intrinsic perceptual limits for current models.

3. The Role of 3D Representations

Double-Edged Sword: Explicit 3D grounding (perfect point clouds) improved performance on depth-sensitive tasks (e.g., +43.3% on Geometric Configuration). However, imperfect 3D reconstruction (via VGGT) was actively harmful, often performing worse than 2D baselines. Noisy reconstructions distorted fine-grained spatial relations, causing models to reason over corrupted scene graphs.

4. The Metacognitive Gap

Premature Commitment: Unlike humans, models commit to answers prematurely with uniformly high confidence, regardless of evidence quality.
Lack of Epistemic Calibration: Humans actively seek falsifying viewpoints and revise beliefs when contradicted. In contrast, models tend to seek confirmation, move in the same direction repeatedly, and anchor to their first impression, failing to update beliefs even when presented with contradictory evidence.

Significance and Claims

The paper claims that ESI-BENCH provides the first rigorous testbed for evaluating the perception-action loop in spatial intelligence, moving beyond passive observation to assess "spatial competence."

Diagnostic Value: The benchmark isolates specific failure modes, revealing that current MLLMs struggle not just with seeing, but with knowing what to look for and when they have seen enough.
Action Selection is Critical: The results suggest that improving spatial intelligence requires better action selection and metacognitive calibration (belief revision) rather than just better visual encoders or more passive data.
Limitations of 3D: The work highlights that simply adding 3D reconstruction is insufficient; the reliability of the reconstruction is paramount, as noisy 3D data can actively mislead reasoning.
Future Directions: The authors posit that closing the gap between human and machine spatial intelligence requires addressing the "metacognitive gap"—specifically, the ability to seek falsifying evidence and revise beliefs under uncertainty.

The paper concludes that while active exploration unlocks emergent strategies, current models lack the epistemic calibration to fully close the perception-action loop, a gap that neither better perception nor more embodied interaction alone can currently bridge.