💻 computer science

How and What to Imagine? Visual Thinking in Unified Multimodal Models for Cross-View Spatial Reasoning

This paper proposes "View Dropout," a training intervention that forces unified multimodal models to utilize intermediate thinking images for cross-view spatial reasoning, demonstrating that panoramic visual thinking combined with this method achieves superior out-of-domain generalization by balancing learnability and informativeness.

Original authors: Qian Yang, Ankur Sikarwar, Huy Le, Le Zhang, Zhuan Shi, Perouz Taslakian, Aishwarya Agrawal

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qian Yang, Ankur Sikarwar, Huy Le, Le Zhang, Zhuan Shi, Perouz Taslakian, Aishwarya Agrawal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: "Talking" vs. "Seeing"

Imagine you are trying to solve a puzzle where you have two photos of a room taken from different corners. You need to tell someone where a specific chair is located relative to a window, even though you can't see both in a single photo.

Current AI models (called Vision-Language Models) are great at looking at a picture and describing it in words. But when it comes to this "cross-view" puzzle, they struggle. Why? Because they try to solve it by talking about the geometry instead of seeing it. They convert the visual puzzle into a list of words (e.g., "The chair is to the left of the table"), and in doing so, they lose the fine-grained spatial details needed to get the answer right.

Humans, on the other hand, don't just talk; we mentally "build" a 3D map in our heads. We imagine walking around the room to see the whole picture. This paper asks: Can we teach AI to do the same thing?

The Proposed Solution: "Thinking in Images"

The researchers tried a method called "Visual Thinking." Instead of just generating a text answer, the AI is forced to generate an intermediate image—a "thinking-image"—before it gives the final answer. This image acts like a mental sketch or a blueprint that bridges the two different camera angles.

They tested three types of these "thinking-images":

Panoramic: Stitching the two views together into one wide, seamless panorama.
Top-Down: Creating a "bird's-eye view" map of the room (like a floor plan).
Point-Matching: Drawing colored dots on the two original photos to show which objects match up.

The Surprise: The AI Was Cheating

Here is the catch: When they first tried this, the AI wasn't actually using the thinking-image. It was just generating a pretty picture as a side effect, then ignoring it to answer the question based on the original photos. It was like a student drawing a diagram in their notebook but then solving the math problem using a calculator they had hidden in their pocket. The diagram was just decoration.

The Fix: "View Dropout" (The Blindfold Trick)

To force the AI to actually use its "thinking-image," the researchers invented a training trick called View Dropout (VDrop).

The Analogy: Imagine you are teaching a student to navigate a city using a map.

Normal Training: You show them the city and the map. They look at the city, guess the answer, and also draw a map. They don't need the map to win.
View Dropout Training: You cover half the city with a blindfold. Now, the student cannot see the city directly to answer the question. They must look at the map they just drew to figure out where the destination is.

In technical terms, during training, the researchers hide a chunk of one of the input photos from the part of the AI that writes the answer. The only way the AI can see that hidden chunk is through the "thinking-image" it generated. This forces the AI to treat the thinking-image as a vital tool, not just a decoration.

The Results: What Works Best?

Once they forced the AI to use the thinking-image, they compared the three types again. They found a trade-off between two things:

Informativeness: How much new spatial information does the image provide?
Learnability: How easy is it for the AI to draw this image correctly?

Top-Down (Bird's-Eye): Very informative (great layout), but hard for the AI to draw perfectly. It often got the angles or object sizes wrong.
Point-Matching: Easy to draw, but not very informative. It just connects dots without showing the full 3D space.
Panoramic (The Winner): This was the sweet spot. It was highly informative (it showed the whole room in one view) and the AI could learn to generate it very reliably.

The Bottom Line

By using View Dropout to force the AI to rely on its own mental sketches, and by choosing the Panoramic style for those sketches, the researchers created a model that is much better at spatial reasoning.

Remarkably, they achieved this with only 8,000 training examples. Other methods tried to solve this by feeding the AI hundreds of thousands of examples, but they failed to force the AI to actually use the visual thinking. This paper proves that the secret isn't just "more data," but the right training trick to make the AI's "imagination" count.

Technical Summary: How and What to Imagine? Visual Thinking in Unified Multimodal Models for Cross-View Spatial Reasoning

Problem Statement

Cross-view spatial reasoning—the ability to infer scene layout, object placement, and geometry from images taken at different viewpoints—remains a significant weakness in Vision-Language Models (VLMs). While VLMs excel at single-image tasks, they perform marginally above chance on cross-view tasks. The authors argue this stems from a representational mismatch: cross-view reasoning is inherently visual, yet VLMs typically reason through language, verbalizing observations into text intermediates that discard the fine-grained geometry required for the task.

Recent "think-with-image" approaches attempt to bridge this gap by generating intermediate visual representations (e.g., 3D reconstructions, depth maps, or predicted camera trajectories). However, prior work indicates that the visual evidence in these generated traces is largely ignored by the model's answer pathway. Under standard supervised fine-tuning (SFT), models generate plausible thinking-images but rely on the original input views to answer, rendering the visual thinking a decorative by-product rather than a functional component of reasoning.

Methodology

1. Unified Multimodal Models (UMMs)

The study utilizes Unified Multimodal Models (specifically BAGEL), which natively support interleaved image-text generation. This architecture allows the model to generate an intermediate visual representation (the "thinking-image") as part of the reasoning chain before producing a textual answer, enabling end-to-end learning without external tools.

2. View Dropout (VDrop)

To address the under-use of visual thinking, the authors propose View Dropout (VDrop), a training-time intervention designed to force the model to rely on the generated thinking-image.

Mechanism: During training, a contiguous region of one input view ( $V_1$ or $V_2$ ) is masked from the attention span of the answer tokens. However, the thinking-image tokens retain full access to the unmasked input views.
Effect: This creates a "shortcut" blockage. Since the answer tokens cannot see the masked region of the input view, they must route their spatial reasoning through the generated thinking-image to recover the missing layout information.
Curriculum: A warmup phase allows the model to learn to generate and use the thinking-image with full input visibility. The masking probability is then linearly annealed to 100%, ensuring the model learns to depend on the thinking-image before the input views are fully obscured.

3. Visual Thinking Strategies

The study compares three distinct types of thinking-images to determine which is most effective, framing the choice as a Learnability–Informativeness (L–I) tradeoff:

Panoramic View: A wide-angle rendering from the observer's pose, unifying the two views into a single visual field.
Top-Down View: A high-angle rendering exposing the global layout and object sides.
Point Matching: The two input views displayed side-by-side with colored markers indicating corresponding objects.

Informativeness refers to how much spatial structure the variant reveals. Learnability refers to how reliably the UMM can produce faithful instances of that variant from data.

4. Training and Evaluation

Data: Training was conducted on 8,000 synthetic samples from Infinigen Indoors, featuring four cross-view question types: Anchor, Counting, Relative Distance, and Relative Direction.
Benchmarks: Evaluation included one in-domain (ID) benchmark (COSMIC) and five real-world out-of-domain (OOD) benchmarks (MMSI-Bench, MindCube, OmniSpatial, STARE-Perspective, BLINK-MultiView).
Causal Verification: A "generate-then-blind" probe was used at inference. The model generates the thinking-image, but the answer span's attention to the thinking-image is masked. A drop in accuracy indicates the thinking-image was causally load-bearing.

Key Results

1. VDrop Makes Visual Thinking Causal

Load-Bearing: Without VDrop, masking the generated thinking-image at inference barely affects accuracy. With VDrop, blinding the thinking-image causes significant accuracy drops (e.g., +12.8% drop on BLINK), confirming the model genuinely relies on the visual intermediate.
Attention Shift: Attention analysis shows VDrop increases the answer tokens' attention share on the thinking-image from ~55% (vanilla) to ~65%, particularly in early and mid-decoder layers.

2. The Learnability–Informativeness Tradeoff

Panoramic: Achieves the best balance. It is highly informative (exposing the joint scene directly) and highly learnable (the model generates faithful renditions). It achieves the highest OOD generalization.
Top-Down: Highly informative but only partially learnable. The model struggles to synthesize the unseen viewpoint with perfect geometric fidelity, leading to mixed results.
Point Matching: Limited by low informativeness (it does not create a new viewpoint) and moderate learnability (precise marker placement is difficult). It yields the smallest gains.

3. Performance Gains

Efficiency: The best configuration (Panoramic + VDrop) trained on only 8K samples achieves a 40.0% average OOD accuracy.
Comparison: This outperforms:
- Vanilla BAGEL (33.3%).
- Strong general-purpose VLMs like Qwen3-VL-8B (37.0%).
- Prior visual-thinking methods trained on significantly more data: ThinkMorph (24K samples, 37.2%) and BAGEL-Zebra-CoT (182K samples, 26.8%).
Non-Visual Baselines: Visual thinking with VDrop significantly outperforms non-visual baselines (No-Think and Text Chain-of-Thought), with Text CoT actually performing worse than direct answering, likely due to inconsistent textual supervision.

Significance and Claims

The paper claims that the primary bottleneck for visual thinking in UMMs is not data scale, but the training signal.

Under-Use is a Failure Mode: The authors identify that standard SFT leads to "decorative" visual thinking where the model ignores its own generated images.
VDrop as a Solution: View Dropout is a simple, architecture-agnostic intervention that forces the thinking-image to become a causally load-bearing component of reasoning.
Representation Matters: Once the model is forced to use the image, the choice of representation becomes critical. The study establishes that panoramic visual thinking is the superior strategy because it is simultaneously informative and learnable, whereas other strategies fail on one or both axes of the L–I tradeoff.
Data Efficiency: The results suggest that with the correct training intervention (VDrop) and representation (Panoramic), high-quality cross-view spatial reasoning can be achieved with significantly less data (8K samples) than previous methods required.

Limitations

The authors note that their findings are validated on a single UMM architecture (BAGEL) and may not transfer to other architectures without further study. Additionally, VDrop forces the use of the thinking-image but does not inherently improve the quality of the generated image; if the generated image is low-fidelity, forcing the model to rely on it yields limited benefits. Future work should combine VDrop with supervision that improves the fidelity of the thinking-image generation itself.