Limits of Imagery Reasoning in Frontier LLM Models

1. Problem Statement

The paper addresses a critical gap in the reasoning capabilities of frontier Multimodal Large Language Models (MLLMs). While these models excel at verbal and symbolic reasoning, they struggle significantly with spatial tasks requiring mental simulation, such as mental rotation (e.g., the Shepard and Metzler task).

• The Core Issue: Current MLLMs treat images as sequences of tokens rather than holistic, transformable 3D states. They lack an internal mechanism for visual state simulation, a deficit the authors metaphorically term "functional aphantasia."
• The Hypothesis: The authors investigate whether equipping an LLM with an external "Imagery Module" (a tool capable of rendering and rotating 3D models) can act as a "cognitive prosthetic" to bridge this gap. They posit that if the burden of maintaining the 3D state is outsourced, the LLM should be able to solve spatial tasks by interacting with the tool.
• The Finding: The study reveals that even with an external tool handling the 3D geometry, frontier models fail to utilize it effectively, suggesting a fundamental lack of visual-spatial primitives required to interface with imagery.

2. Methodology: The Prosthetic Imagery Architecture

The authors designed a dual-module agentic loop to test this hypothesis on the SpatialViz 3D rotation benchmark.

• System Components:
  1. Reasoning Module: A frontier MLLM (primarily GPT-5.2) that acts as the agent. It does not manipulate geometry directly but issues rotation commands based on visual feedback.
  2. Imagery Module: A Python/PyVista-based tool that maintains a persistent 3D state for objects. It executes rotation commands (e.g., "left:30", "up:15") relative to the current camera view and renders 2D snapshots for the LLM.
• Workflow:
  • The system initializes with a calibrated camera pose matching the benchmark prompt.
  • The LLM receives the problem statement, the original image, and the latest snapshot from the Imagery Module.
  • The LLM issues rotation commands to align the object with a target view or to inspect specific features.
  • The loop continues iteratively until the LLM selects the correct answer (identifying which option cannot be obtained by rotation).
• Experimental Conditions:
  1. Reset Enabled: Allowed the model to reset objects to canonical views (reducing the task to 2D image matching).
  2. Incremental Rotation with 360° Hint: Provided a full 360° trajectory to allow simple image matching without planning.
  3. Incremental Rotation Only (Main Setup): Required the model to perform step-by-step rotations to find a geometric match.

3. Key Contributions

1. Empirical Evidence of "Functional Aphantasia": The paper provides concrete evidence that current frontier LLMs lack the internal mechanisms for visual simulation. Even when the 3D state is perfectly maintained by an external tool, the LLM cannot "think" in images.
2. Identification of Missing Primitives: The authors isolate two specific foundational deficits preventing spatial reasoning:
   • Low-level Sensitivity: Inability to extract spatial signals like depth, motion (interpreting two images as a temporal sequence), and short-horizon dynamic prediction.
   • Contemplative Reasoning: Inability to dynamically shift visual focus or "contemplate" an image. Instead, models default to symbolic, word-based associations (e.g., counting cubes verbally) rather than engaging in geometric matching.
3. Ablation of Tool Usage: The study demonstrates that simply attaching a tool is insufficient. The failure lies in the interface between the reasoning module and the tool, not the tool's capability.

4. Results

• Performance Ceiling: In the main experimental setup (Incremental Rotation Only), the best-performing model (GPT-5.2) achieved an accuracy of 50%–62.5%, far below human performance (~79%) and significantly lower than the theoretical maximum if the tool were used correctly.
• Condition 1 (Reset): When the task was reduced to 2D matching via a reset command, accuracy jumped to 85–97.5%. This proves the models can solve the logic if the spatial simulation burden is removed, confirming the bottleneck is the simulation/rotation process.
• Condition 2 (360° Hint): Even with a full 360° view provided to bypass planning, accuracy remained low (52.5%). The models failed to identify matching frames in a grid of images, reverting to verbal inference.
• Core Capability Tests:
  • Rotation Detection: When asked to identify the direction/angle of a small rotation between two images, models failed consistently (e.g., predicting "Right" when the object rotated "Left").
  • State Prediction: When asked to generate an image of an object after a rotation, models failed completely, outputting the original image unchanged.
  • Specialized Models: A specialized vision transformer (VGGT) successfully predicted camera extrinsics (yaw/pitch/roll) with high accuracy, proving that the Transformer architecture itself is not the bottleneck, but rather the training strategy and lack of supervised spatial data in generalist MLLMs.

5. Significance and Conclusion

The paper concludes that the limitation of current LLMs in spatial reasoning is not merely a lack of a "rendering engine" but a fundamental architectural deficit in visual-spatial primitives.

• The "Prosthetic" Failure: An external imagery module cannot function as a cognitive prosthetic if the host (the LLM) lacks the "nerves" (perceptual primitives) to interpret the data it receives.
• Symbolic Bias: MLLMs exhibit a strong bias toward symbolic, text-based reasoning. They treat images as static tokens to be described rather than dynamic states to be manipulated. They fail to "contemplate" visual features or simulate motion.
• Future Directions: The authors argue that future research must move beyond simple tool-use or parallel imagery branches. Instead, it must focus on building foundational architectures that intrinsically support:
  • Spatial awareness and depth extraction.
  • Continuous state tracking (interpreting sequences of images as motion).
  • Contemplative, attention-driven visual reasoning.

In summary, the paper establishes that tool-augmentation alone is insufficient to solve spatial reasoning problems in frontier LLMs because the models lack the foundational ability to perceive and predict dynamic visual states.