💬 NLP

Unleashing Spatial Reasoning in Multimodal Large Language Models via Textual Representation Guided Reasoning

This paper introduces TRACE, a prompting method that enhances the 3D spatial reasoning capabilities of Multimodal Large Language Models by guiding them to generate structured text-based allocentric representations of egocentric video environments, resulting in consistent performance improvements across diverse benchmarks and model backbones.

Original authors: Jiacheng Hua, Yishu Yin, Yuhang Wu, Tai Wang, Yifei Huang, Miao Liu

Published 2026-03-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiacheng Hua, Yishu Yin, Yuhang Wu, Tai Wang, Yifei Huang, Miao Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a house while holding a camera, filming everything you see. You are looking at a coffee cup, then a lamp, then a door. This is an egocentric video (a "self-centered" view).

Now, imagine someone asks you a tricky question: "If I were standing by the trash can and facing the telephone, would the cup be to my front-left or back-right?"

For a human, this is easy. We don't just look at the video frame-by-frame. We instantly build a mental map of the whole room in our heads. We know where the walls are, where the furniture sits relative to each other, and we can mentally "teleport" ourselves to the trash can to answer the question.

The Problem:
Current AI models (Multimodal Large Language Models, or MLLMs) are like a person who is only allowed to look at one photo at a time. They are great at describing what's in the picture ("I see a red cup"), but they are terrible at building that 3D mental map. They get lost in the details and can't figure out the spatial relationships, much like a tourist who knows what a street looks like but doesn't know how to get from the hotel to the museum.

The Solution: TRACE
The authors of this paper created a new method called TRACE (Textual Representation of Allocentric Context from Egocentric Video).

Think of TRACE as a translator that turns a confusing, moving video into a structured "cheat sheet" or a blueprint before the AI tries to answer the question.

Here is how TRACE works, using a simple analogy:

The Analogy: The Detective's Case File

Imagine the AI is a detective trying to solve a crime scene based on a shaky security video.

The Old Way (Direct Prompting): The detective watches the video and immediately tries to guess the answer. They might say, "I think the cup is on the left!" but they are just guessing based on a single blurry frame. They often get it wrong.
The TRACE Way: Before guessing, the detective is forced to write a Case File (the TRACE) that organizes the scene. This file has three specific sections:
- Section 1: The Room Blueprint (Meta-Context)
  - Analogy: The detective draws a simple map of the room. "It's a rectangular office. North is the window wall. The origin point (0,0) is the door."
  - Why it helps: It gives the AI a fixed coordinate system so it doesn't get confused by the camera spinning around.
- Section 2: The Walkthrough Log (Camera Trajectory)
  - Analogy: The detective writes down their steps: "At 0 seconds, I stood at the door facing North-West. At 5 seconds, I walked 2 meters forward and turned right to look at the desk."
  - Why it helps: It tracks movement. The AI knows exactly where the "observer" was at every moment, turning a chaotic video into a clear path.
- Section 3: The Object Registry (Entity Registry)
  - Analogy: The detective lists every item found, like a grocery list but with GPS coordinates. "Trash Can: 2 meters left of the door. Cup: 1 meter left of the phone."
  - Why it helps: Instead of just "seeing" a cup, the AI now has a precise list of where everything is relative to everything else.

Why This Works

Once the AI generates this "Case File" (the text description), it doesn't have to guess anymore. It can simply read the blueprint to answer the question.

Question: "Is the cup to the left of the phone?"
TRACE Answer: "Look at the registry. The phone is at [2, 2]. The cup is at [1, 2]. Therefore, the cup is to the left."

The Results

The researchers tested this on two major benchmarks (VSI-Bench and OST-Bench) using various AI models.

The Result: TRACE consistently beat all other methods. It helped even the smaller, less powerful AI models perform much better.
The Takeaway: By forcing the AI to stop and "draw a map" (in text form) before answering, we unlock its ability to understand 3D space. It's like giving a student a graph paper and a ruler before asking them to solve a geometry problem, rather than just asking them to guess the answer in their head.

In short: TRACE teaches AI to stop staring at the pixels and start building a mental map, turning a confusing video into a clear, organized story that the computer can actually reason about.

1. Problem Statement

Multimodal Large Language Models (MLLMs) currently struggle with 3D spatial reasoning tasks, particularly when processing egocentric (first-person) video inputs. Despite being trained on massive video datasets, these models often fail to construct structured abstractions of the 3D environment. Instead of building hierarchical representations of spatial relations, they tend to:

Over-fixate on 2D visual signals.
Rely on shortcut correlations from implicit spatial cues.
Fail to maintain consistent internal 3D representations, leading to errors in tasks requiring holistic scene understanding (e.g., determining object locations relative to a moving camera).

Existing solutions either require massive supervised fine-tuning (limiting scalability) or incorporate complex geometric/stereo modalities (increasing system complexity and restricting applicability to off-the-shelf models). The authors pose the question: Can MLLMs be guided to explicitly construct and reason over structured allocentric (world-centered) representations of 3D spatial environments derived solely from 2D visual observations?

2. Methodology: TRACE

The authors propose TRACE (Textual Representation of Allocentric Context from Egocentric Video), a prompting method that induces MLLMs to generate a structured text-based representation of the 3D environment as an intermediate reasoning trace.

Inspired by cognitive theories of human spatial reasoning (transforming visual input into hierarchical object/relationship representations), TRACE forces the model to output a specific schema before answering the final question. The representation $G$ is defined as a tuple $\langle M, T, E \rangle$ :

Key Components of TRACE:

Meta Context ( $M$ ):
- Establishes a Room-Aligned Coordinate System.
- Defines the room topology (e.g., "Rectangular Office").
- Aligns the grid axes (e.g., $+Y$ aligned with the longest visible wall) rather than the camera's initial heading.
- Records the observer's initial heading relative to this global grid.
Camera Trajectory ( $T$ ):
- Reconstructs the observer's path as a discrete sequence of steps.
- For each step, it logs: timestamp, estimated 2D position $[x, y]$ , facing direction (cardinal points), and action context.
- This allows the model to traverse a "static map" rather than relying on transient visual memory.
Entity Registry ( $E$ ):
- Maintains a structured list of all observed objects with unique IDs (e.g., chair_01).
- For each entity, it records:
  - Temporal Stamping: First seen time.
  - Visual Signature: Brief appearance description to disambiguate similar objects.
  - Metric Estimation: Estimated 2D coordinates $[x, y]$ in meters relative to the grid origin.
  - Spatial Relations: Natural language descriptions of relations to nearby anchors.
  - Strict Serialization: Objects are listed individually to ensure accurate counting and localization.

Inference Mechanism

The inference process is a single-pass generation where the MLLM first generates the structured TRACE (the "Spatial Descriptor") and then uses this text-based "spatial cache" to answer the query (the "Reasoning Parser"). This acts as a specialized Chain-of-Thought (CoT) that explicitly loads the context window with geometric constraints before deriving the final answer.

3. Key Contributions

Novel Prompting Strategy: Introduced TRACE, the first prompting-based method that enables off-the-shelf MLLMs to perform structured 3D spatial reasoning by generating intermediate allocentric text representations.
Structured Representation Design: Defined a rigorous schema (Meta Context, Trajectory, Entity Registry) that bridges the gap between 2D video inputs and 3D spatial reasoning, forcing the model to resolve spatial relations into geometric constraints.
Comprehensive Evaluation: Validated the approach on two major benchmarks (VSI-Bench and OST-Bench) across diverse model backbones (Gemini 3 Pro, Qwen2.5-VL, MiMo-VL, o3, GLM-4.5V) and parameter scales.
Diagnostic Insights: Provided decompositional analyses revealing that current MLLMs often fail due to weaknesses in the "Spatial Descriptor" (visual perception of 3D structure) rather than just the reasoning parser, and that explicit geometric grounding is crucial for smaller models.

4. Experimental Results

The paper reports extensive experiments demonstrating that TRACE consistently outperforms existing prompting strategies (Direct, CoT, Tree-of-Thought, Least-to-Most, and Cognitive Map).

Performance Gains:
- VSI-Bench: TRACE achieved state-of-the-art results. For example, on Gemini 3 Pro, it improved the average score by +7.54% over the Direct baseline and +0.43% over the previous best (Cognitive Map). On Qwen2.5-VL-72B, it improved by +3.10%.
- OST-Bench: TRACE showed consistent gains, particularly on the open-source MiMo-VL-7B model (+2.4% over Direct), proving its effectiveness for smaller, compact models where explicit geometric grounding is more critical than general linguistic reasoning.
Comparison with Other Methods:
- TRACE outperformed the Cognitive Map approach (Yang et al., 2025b) by 9.7% on VSI-Bench (text-only setting), demonstrating that the granularity of TRACE (individual entities with coordinates) is superior to coarse 10x10 grids.
- It also outperformed Spatial Captioning methods by 3.53%.
Ablation Studies:
- Removing the Entity Registry caused the largest performance drop (-5.24%), highlighting the importance of object-level grounding.
- Removing Trajectory information caused a -1.92% drop, affecting distance and route planning tasks.
Generalization: The method showed robust performance gains across different indoor scene datasets (ARKitScenes, ScanNet, ScanNetPP), indicating it does not overfit to specific environment types.
Token Efficiency: On compact models (MiMo), TRACE was more token-efficient than ToT and LtM while delivering better accuracy.

5. Significance and Conclusion

This work addresses a critical bottleneck in multimodal AI: the inability of current models to reason about 3D space from 2D video. By shifting the paradigm from direct pixel-to-answer mapping to text-guided intermediate reasoning, TRACE provides a simple, scalable, and highly effective interface for enhancing spatial intelligence in MLLMs.

Key Takeaways:

Cognitive Alignment: The approach aligns with human cognitive processes by forcing the model to construct an allocentric (world-centered) mental model before answering.
Model Agnostic: It works effectively across proprietary and open-source models of varying sizes without requiring architectural changes or massive retraining.
Future Direction: The authors suggest that this textual representation could serve as a data engine for training future spatially-aware models and propose future work on dynamic, streaming TRACE frameworks for real-time agent state tracking.

In summary, TRACE demonstrates that structured textual representations are a powerful intermediate step to "unleash" the latent 3D spatial reasoning capabilities of existing Multimodal Large Language Models.