💻 computer science

CVT-Bench: Counterfactual Viewpoint Transformations Reveal Unstable Spatial Representations in Multimodal LLMs

This paper introduces CVT-Bench, a diagnostic benchmark revealing that despite high single-view accuracy, state-of-the-art multimodal LLMs exhibit unstable spatial representations under counterfactual viewpoint changes, a limitation that is mitigated by using more structured input representations.

Original authors: Shanmukha Vellamcheti, Uday Kiran Kothapalli, Disharee Bhowmick, Sathyanarayanan N. Aakur

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shanmukha Vellamcheti, Uday Kiran Kothapalli, Disharee Bhowmick, Sathyanarayanan N. Aakur

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are standing in a room with a table. On the table, there is a red cup, a blue ball, and a green box. You look at them and say, "The cup is to the left of the ball."

Now, imagine you walk around the table to the opposite side. From your new spot, the cup is now to the right of the ball. A human brain handles this switch instantly. You don't need to take a new photo; you just mentally "spin" the room in your head and update the map.

This paper, CVT-Bench, is like a "driver's license test" for AI, but instead of driving a car, it's testing how well AI can do this mental spinning.

The Big Problem: The AI's "Amnesia"

The researchers found that while modern AI models (Multimodal Large Language Models) are amazing at describing what they see in a single photo, they are terrible at imagining what that scene looks like from a different angle.

Think of these AIs like a tourist with a camera but no map.

The Tourist (AI): If you show them a photo of a park, they can perfectly describe the trees and benches.
The Test: If you ask, "Okay, if I walk 90 degrees to the left, where are the benches now?" the tourist panics. They might say, "I don't know, I only have this one photo!" or they might guess wildly, mixing up left and right.

The paper calls this "Counterfactual Viewpoint Transformation." It's a fancy way of asking: "If I moved, how would the world change?"

The Experiment: The "Mental Gym"

The researchers built a gym called CVT-Bench to test this.

The Scene: They created 100 digital rooms filled with simple shapes (cubes, spheres, cylinders).
The Test: They showed the AI a picture of the room. Then, they asked questions like, "If I rotate the camera 180 degrees (turn around completely), is the red cube in front of or behind the blue sphere?"
The Catch: The AI wasn't allowed to see the new angle. It had to rely entirely on its internal memory of the room to figure it out.

They also tested the AI in two modes:

The "One-Off" Mode: Asking one question about one room. (The AI does okay here).
The "Marathon" Mode: Asking the AI to solve 20 different rooms in a row without a break. (The AI crashes hard here).

The Results: The "Brittle" Brain

The results were surprising and a bit scary for the future of robotics:

The "U-Turn" Failure: When the AI had to imagine a 90-degree turn (a sharp corner), it often got completely confused. It would mix up "left" and "right" like a person who is dizzy.
The "Long Memory" Problem: When the AI had to remember many scenes in a row (the Marathon Mode), its performance dropped off a cliff. It's like trying to remember a grocery list while someone is shouting numbers at you; the more you try to hold, the more you drop.
The "Text vs. Image" Surprise: When the researchers gave the AI the scene as a text description (e.g., "Cube is at X, Sphere is at Y") instead of a picture, the AI did better. This suggests the AI isn't failing because it can't "see" the image, but because it can't build a stable 3D map in its mind when looking at pixels.

The "Magic Glasses" (Structured Data)

The researchers tried giving the AI "training wheels." Instead of just a picture or a paragraph, they gave the AI a Scene Graph—a structured list that explicitly says, "Cube is LEFT of Sphere."

Analogy: It's like giving a tourist not just a photo, but a GPS map with arrows drawn on it.
Result: The AI got much more stable. It didn't get as dizzy. This proves that if we want AI to be good at spatial reasoning, we can't just throw pictures at them; we need to help them organize the information into a clear structure first.

Why Should You Care?

You might think, "So what? It's just a toy with shapes." But this is the foundation for robots and self-driving cars.

The Robot Butler: Imagine a robot in your kitchen. You say, "Hand me the mug on the left of the toaster." The robot needs to know where the mug is from your perspective, not just its own. If the robot can't mentally rotate the scene, it will grab the wrong cup or knock things over.
The Self-Driving Car: If a car sees a pedestrian, it needs to predict where that pedestrian will be if the car turns a corner. If the car's "brain" is unstable, it might think the pedestrian is safe when they are actually in danger.

The Bottom Line

The paper concludes that current AI is "view-dependent." It relies too much on the specific photo it's looking at right now. It hasn't learned to build a stable, 3D "world model" that it can spin around in its head.

To make AI truly smart and safe for the real world, we need to teach it how to keep its balance when the world spins around it, not just how to describe the world when it's standing still.

1. Problem Statement

While Multimodal Large Language Models (MLLMs) demonstrate high accuracy in single-view spatial reasoning tasks (e.g., identifying objects and describing their relative positions in a static image), it remains unclear whether they possess stable, viewpoint-invariant spatial representations.

Current evaluations are largely episodic, assessing reasoning within a single scene and fixed viewpoint. However, real-world applications (robotics, autonomous driving) require agents to:

Infer how spatial relationships transform under counterfactual viewpoint changes (e.g., "If I move 180° around this table, where is object A relative to object B?") without new visual input.
Maintain consistent spatial states across extended sequential interactions involving multiple scenes.

The authors hypothesize that current MLLMs rely heavily on view-dependent cues rather than internalizing a robust 3D scene representation, leading to instability when the observer's perspective changes or when context length increases.

2. Methodology: CVT-Bench

The authors introduce CVT-Bench, a controlled diagnostic benchmark designed to isolate spatial reasoning capabilities from object recognition and perception.

A. Dataset Construction

Source: Procedurally generated scenes based on the CLEVR environment, ensuring precise 3D coordinates and unambiguous object identities.
Scale: 100 synthetic scenes (50 sparse: 3–5 objects; 50 dense: 7–10 objects) with varying occlusion levels (5%–70%).
Queries: Approximately 6,000 relational queries.
Viewpoints: Queries are generated for an observed anchor viewpoint ( $0^\circ$ ) and 9 counterfactual viewpoints (rotations of $45^\circ$ to $360^\circ$ and a top-down allocentric view).
Task: Given a scene from viewpoint $v_0$ , the model must predict spatial relations (Left, Right, Front, Behind) between object pairs under a hypothetical rotation $\theta$ , without receiving visual input from the new viewpoint.

B. Evaluation Protocols

Episodic Evaluation: Each scene is presented independently. Measures instantaneous reasoning ability.
Sequential Evaluation: Multiple independent scenes are concatenated into a single long-context prompt. Measures the ability to maintain spatial state without interference from previous scenes (context accumulation).
Input Representations: To diagnose the source of failure, models are tested across three modalities:
- Visual: Raw rendered images with object tags.
- Textual: Structured natural language descriptions with bounding boxes (removes visual perception).
- Scene Graph: Explicit relational graphs derived from ground-truth coordinates (provides maximum structural supervision).

C. Metrics

F1 Score: Used for multi-label classification of spatial relations.
Spatial Survival Rate: A novel metric measuring the probability that a model maintains correct predictions consecutively as interaction depth increases.
Cycle Consistency: Checking if the model's answer for $0^\circ$ matches its answer for $360^\circ$ (which are physically identical).

3. Key Contributions

CVT-Bench Benchmark: A diagnostic suite specifically targeting counterfactual viewpoint reasoning and visual perspective-taking, distinct from standard VQA benchmarks.
Unified Protocol: A framework distinguishing between episodic (single-scene) and sequential (multi-scene) reasoning to analyze state maintenance.
Representation-Controlled Evaluation: Systematic comparison of visual, textual, and graph-based inputs to isolate whether failures stem from perception, representation structure, or reasoning logic.
Large-Scale Findings: Empirical evidence showing that high single-view accuracy masks significant instability in spatial reasoning under rotation and long contexts.

4. Key Results

The study evaluated state-of-the-art models (GPT-5.2, Gemini 3.1 Pro, Qwen 3.5, Kimi 2.5) and found:

Viewpoint Instability: While models achieve >90% F1 at the anchor ( $0^\circ$ $0^{\circ}$ ) view, performance degrades systematically under counterfactual rotations.
- Orthogonal Drops: Significant performance cliffs occur at $90^\circ$ and $270^\circ$ (orthogonal angles), suggesting models use brittle heuristics rather than smooth internal rotation.
- Top-View Robustness: Paradoxically, top-down (allocentric) queries often perform better than in-plane rotations, suggesting models struggle more with egocentric horizontal transformations.
Sequential Degradation: In sequential settings (BS=20), performance drops significantly for both anchor and rotated views.
- Context Drift: Accuracy decreases as the prompt depth increases (later scenes are answered less accurately), indicating a failure to maintain a coherent spatial state over long contexts.
- Survival Rate Collapse: The "Spatial Survival Rate" (consecutive correct answers) drops sharply, revealing that models cannot sustain relational consistency over extended interactions.
Representation Impact:
- Text vs. Image: Text-only inputs often show less degradation than raw images, implying that visual grounding introduces noise or instability.
- Scene Graphs: Structured scene graphs improve stability in some models but do not fully solve the problem, indicating that the issue is not just input format but the internal reasoning mechanism.
Failure Modes:
- Viewpoint Inversion: Systematic confusion of Left/Right or Front/Behind at specific angles.
- Cross-Scene Interference: In sequential tasks, models confuse relations from previous scenes with current ones.
- Cycle Inconsistency: Models often give different answers for $0^\circ$ and $360^\circ$ queries despite identical physical states.

5. Significance and Conclusion

Overestimation of Robustness: The paper concludes that standard single-view spatial accuracy overestimates the robustness of MLLMs. High performance on static images does not imply the model has learned a stable, transformable 3D world model.
Mechanism of Failure: The instability suggests MLLMs rely on view-dependent reasoning (associating objects with the current image features) rather than maintaining an internal, viewpoint-invariant spatial state.
Implications for Embodied AI: For robotics and autonomous systems, where agents must reason about the world from different perspectives and over time, current MLLMs are unreliable without explicit mechanisms for relational state representation.
Future Directions: The authors suggest that improving spatial reasoning requires architectural changes to explicitly model relational states and perspective transformations, rather than relying solely on end-to-end perception-to-action pipelines.

In summary, CVT-Bench reveals a critical gap in current MLLMs: they are excellent at describing "what they see" but struggle to reason about "what the world looks like from another angle" or "what the world looked like previously," highlighting a fundamental lack of stable spatial cognition.