Visuospatial Perspective Taking in Multimodal Language Models

This paper reveals that multimodal language models exhibit significant deficits in Level 2 visuospatial perspective-taking, struggling to inhibit their own viewpoint to adopt another's, which highlights critical limitations for their deployment in social and collaborative settings.

The Gist

Imagine you are playing a game of "Simon Says" with a very smart robot friend. You are standing back-to-back. You see a red ball on your left and a blue ball on your right. Your robot friend, however, is facing the opposite direction. To them, the red ball is on their right, and the blue ball is on their left.

If you say, "Pick up the red ball on your left," a human would instantly realize: "Wait, from my perspective, the red ball is on the right. But from the robot's perspective, it's on the left. I need to flip my brain to see what they see."

This paper asks a simple but profound question: Can modern AI models do this "brain flip"?

The researchers tested the latest, most advanced Multimodal Language Models (MLMs)—the kind of AI that can see pictures and read text—to see if they truly understand Visuospatial Perspective Taking (VPT). This is the ability to imagine the world not just from your own eyes, but from someone else's.

Here is the breakdown of their findings, using some everyday analogies:

1. The Two Levels of "Seeing"

The researchers looked at two types of perspective-taking, which they call Level 1 and Level 2.

• Level 1 (The "Can You See It?" Test): This is like asking, "Is the cat behind the sofa?" If you can see the cat, and the other person is looking at the sofa, you know they can see it too. If the cat is hidden behind a wall, you know they can't.

  • The Result: The AI was pretty good at this. It could usually tell if an object was visible or hidden. It's like a security guard who can easily spot if a door is open or closed.

• Level 2 (The "What Does It Look Like?" Test): This is the hard part. It's not just about seeing the object; it's about knowing how it looks to them. If you see a number "6" on the floor, and your friend is standing opposite you, they see a "9".

  • The Result: The AI struggled massively here. It's like a robot that can see the number "6" but cannot mentally rotate its head to realize, "Oh, if I were standing there, that would look like a 9."

2. The "M-Shaped" Failure

The researchers spun the AI's "head" (the figure in the image) in a full circle, from 0 degrees (facing you) to 180 degrees (facing away).

• The Human Way: Humans get slightly slower and make a few more mistakes as the angle gets weird, but we generally get it right.
• The AI Way: The AI showed a strange "M-shaped" pattern.
  • When the figure faced exactly the same way as the AI (0°), it was perfect.
  • When the figure faced exactly the opposite way (180°), it was also surprisingly good.
  • But in the middle? When the figure was turned slightly to the side (45°, 90°, 135°), the AI's performance crashed.

The Analogy: Imagine a mirror. If you stand directly in front of it, you see yourself clearly. If you stand directly behind it, you know you can't see yourself. But if you stand at a weird angle, the AI gets confused. It seems to rely on a simple trick: "If they are opposite me, I just swap left and right." But if they are just slightly turned, that trick fails, and the AI gets lost.

3. The "Director" Game

To test this in a real-world scenario, they used a game called the Director Task.

• The Setup: A human "Director" stands on one side of a shelf with a grid of items. Some items are hidden from the Director's view by opaque backs. The AI (the "Player") must pick the item the Director asks for.
• The Trap: The Director says, "Pick the smallest star." But there are two stars. One is small and visible to the AI but hidden from the Director. The other is small and visible to both.
• The Failure: The AI often picked the star that it could see, forgetting that the Director couldn't see it. It couldn't "inhibit" its own privileged view to adopt the Director's limited view.

4. Why Does This Matter?

You might think, "So the AI is bad at spinning numbers around. Who cares?"

The paper argues that this is a critical flaw for the future. We are starting to use AI as teachers, therapists, and collaborative partners.

• If you ask a robot to "hand me the cup on the left," and it doesn't understand that its left is your right, it will hand you the wrong cup.
• If a virtual therapist doesn't understand what you can see in your room versus what they can see on a screen, they might give bad advice.

The Bottom Line

The paper concludes that while these AI models are incredibly smart at reading and recognizing images, they are not actually "thinking" about other people's viewpoints.

They are like a very talented actor who has memorized a script. If the script says "swap left and right," they do it. But if the situation is complex and requires genuinely imagining being someone else, they stumble. They haven't learned to "walk a mile in someone else's shoes"; they've just learned to guess what the shoes look like from the outside.

In short: Current AI is great at seeing the world, but it's still learning how to see the world through someone else's eyes. Until it masters this, we need to be careful about trusting it in complex social or collaborative situations.

Technical Summary

1. Problem Statement

As Multimodal Language Models (MLMs) are increasingly deployed in social, collaborative, and embodied settings (e.g., as teachers, therapists, or robotic assistants), their ability to understand and reason about the perspectives of others—specifically Visuospatial Perspective Taking (VPT)—is critical.

Current evaluations of AI "Theory of Mind" (ToM) rely heavily on text-based vignettes or static scene recognition. These methods have significant limitations:

• Superficial Learning: Models may exploit shallow associations in training data rather than reasoning about causal mental states.
• Lack of Context: Text vignettes remove the inferential challenges of real-world social interaction where perspective must be inferred from visual context and occlusion.
• Underexplored VPT: While Level 1 VPT (determining what someone sees) is studied, Level 2 VPT (determining how something appears to someone else, requiring mental rotation and inhibition of one's own ego-centric view) remains largely untested in frontier MLMs.

The paper argues that without robust VPT capabilities, MLMs cannot safely or effectively participate in collaborative tasks requiring shared attention, referential communication, or joint action.

2. Methodology

The authors adapted two established human cognitive psychology tasks into a rigorous, procedurally generated benchmark for MLMs. They tested four frontier models from the OpenAI family: GPT-4o, GPT-4o-mini, o3, and o4-mini (the latter two being reasoning-optimized models).

A. The Rotating Figure Task (RFT)

This task isolates VPT across angular disparities (0° to 359°) and distinguishes between visual and spatial content.

• Setup: A top-down image shows a figure in a room with a reversible symbol (e.g., "6" or "9") on the floor.
• Conditions:
  • Controls: Test baseline perception (symbol identification, figure orientation).
  • Test 1 (Level 1 VPT): Determine if the figure can see the symbol (Visibility).
  • Test 2 (Level 2 VPT): Determine how the symbol appears to the figure (e.g., is a "6" seen as a "9"?) and its spatial location (left/right) from the figure's perspective.
  • Test 3 (Integrated Level 2 VPT): Simultaneously reason about visual appearance and spatial location (e.g., "What is on the figure's right?").
• Manipulation: The angular disparity between the viewer and the figure is varied systematically to test performance at shared (0°), intermediate (90°), and opposite (180°) perspectives.

B. The Director Task (DT)

This task assesses VPT within a functional, referential communication paradigm.

• Setup: A 4x4 grid of items is viewed by the model. A "Director" stands on the opposite side. Some grid cells have opaque backs, occluding items from the Director's view.
• Task: The model must select a specific item based on the Director's instructions (e.g., "Select the smallest star").
• Critical Constraints:
  • Visual Perspective: The model must ignore items the Director cannot see (occlusion).
  • Spatial Perspective: The model must resolve relative adjectives (e.g., "leftmost" vs. "rightmost") based on the Director's inverted perspective, not its own.
• Comparison: The task was run in both Image-based and ASCII-based (text-only) formats to distinguish between visual processing deficits and genuine perspective-taking failures.

3. Key Contributions

1. Systematic Benchmarking: The study moves beyond single-model, single-trial evaluations by testing a broad range of models with thousands of procedurally generated trials (27k for RFT, 16k for DT), ensuring statistical power and reducing contamination.
2. Differentiation of VPT Levels: It rigorously separates Level 1 (visibility) from Level 2 (appearance/mental rotation) and visual from spatial components, revealing that models struggle disproportionately with Level 2.
3. Identification of Heuristic Reliance: The paper provides evidence that models often rely on simple "mirroring" heuristics (e.g., swapping left/right only at 180°) rather than performing genuine mental rotation, leading to specific failure patterns (e.g., "M-shaped" performance curves).
4. Cross-Modal Validation: By comparing image and ASCII versions of the Director Task, the authors demonstrate that VPT deficits are cognitive/representational limitations, not merely artifacts of poor image processing.

4. Key Results

A. Rotating Figure Task Findings

• Level 1 VPT (Visibility): Models generally performed well but showed specific blind spots. For instance, GPT-4o and o4-mini struggled with right-facing rotations, while o3 produced high rates of invalid responses (hallucinations).
• Level 2 VPT (Appearance & Spatial):
  • Non-Reasoning Models (GPT-4o, GPT-4o-mini): Showed monotonic declines in accuracy as angular disparity increased. GPT-4o-mini failed almost completely at 180° (inverted perspective) for spatial questions.
  • Reasoning Models (o3, o4-mini): Exhibited "M-shaped" performance profiles. They performed well at 0° (shared) and 180° (opposite) but suffered significant accuracy drops at intermediate angles (45°–135°).
  • Interpretation: The M-shape suggests models use a simple heuristic (e.g., "if 180°, swap left/right") rather than a generalizable mental rotation mechanism.
• Integrated VPT (Test 3): When visual and spatial demands were combined, all models dropped to near-chance or floor performance as perspectives diverged, indicating a failure to integrate multiple perspective-taking constraints.

B. Director Task Findings

• Visual vs. Spatial Costs: Both visual (occlusion) and spatial (left/right reversal) perspective-taking reduced accuracy, but spatial costs were significantly larger.
• Model Differences:
  • o3: Showed robust performance on spatial tasks when perspectives were fully opposite (180°) but struggled when visual occlusion was added.
  • Other Models: Showed severe deficits in both domains.
• Error Analysis: When models failed on "leftmost/rightmost" instructions from the Director's POV, they typically selected items on the opposite side of the grid relative to the correct answer, confirming they failed to inhibit their own ego-centric perspective.
• ASCII vs. Image: Performance patterns were consistent across both modalities, proving the failure is not due to visual recognition errors but a lack of spatial reasoning capabilities.

5. Significance and Implications

• Limitations of Current AI: The study exposes a critical gap in current MLMs: they lack the representational capacity for Level 2 VPT (mental rotation and ego-inhibition). They can recognize what is visible but struggle to simulate how the world looks from another's viewpoint, especially at intermediate angles.
• Safety and Deployment: In collaborative settings (e.g., a robot assisting a human), failure to adopt the user's perspective can lead to dangerous errors (e.g., handing an object to a person who cannot see it, or misinterpreting spatial instructions).
• Mechanistic Insight: The "M-shaped" performance curve suggests that reasoning models are applying brittle heuristics rather than learning a general theory of spatial perspective. This challenges the notion that "Chain-of-Thought" or "Reasoning" models have solved social cognition; they may simply be better at applying specific rules for edge cases (0° and 180°) while failing in the continuous middle ground.
• Future Directions: The authors call for architectural changes to support genuine mental rotation and the development of benchmarks that move beyond text-based vignettes to dynamic, multimodal, and interactive scenarios.

In conclusion, while MLMs show promise in basic perception, they currently lack the robust, flexible visuospatial perspective-taking abilities required for safe and effective human-AI collaboration.