🤖 AI

Planning with the Views via Scene Self-Exploration

This paper introduces ViewSuite, a 3D benchmark revealing that Vision-Language Models struggle to compose view-action transformations for multi-turn planning, and proposes an iterative self-exploration framework with view graph distillation that significantly boosts planning performance by overcoming sparse rewards and surpassing leading models like GPT-5.4 Pro.

Original authors: Kangrui Wang, Linjie Li, Zhengyuan Yang, Shiqi Chen, Zihan Wang, Li Fei-Fei, Jiajun Wu, Leonidas Guibas, Lijuan Wang, Manling Li

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kangrui Wang, Linjie Li, Zhengyuan Yang, Shiqi Chen, Zihan Wang, Li Fei-Fei, Jiajun Wu, Leonidas Guibas, Lijuan Wang, Manling Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are in a giant, unfamiliar house with thousands of rooms, and someone hands you a photo of a specific corner in one of those rooms. Your goal is to walk through the house, looking around, until you find that exact spot and say, "I'm here."

This is the challenge the paper VIEWSUITE tackles, but instead of a human, they are testing AI Vision-Language Models (VLMs)—smart computers that can see pictures and read text.

Here is the story of what they found and how they fixed it, explained simply:

1. The Problem: The "One-Step" Genius vs. The "Long-Hike" Struggle

The researchers discovered that these AI models are actually pretty good at one-step thinking.

The Test: If you show the AI a picture of a room and say, "If I turn right twice, what will I see?" the AI can usually guess the new picture correctly.
The Failure: But if you ask, "Walk through the house to find this specific photo," the AI gets lost immediately. It knows how to turn, but it can't plan a route. It's like a person who knows how to take a single step but gets dizzy and confused if asked to plan a 10-step journey to a destination.

The paper calls this the "Planning Gap." The AI understands the rules of moving (turning left, moving forward) but fails to string them together into a long-term plan.

2. The Environment: A Digital "Point Cloud" Maze

To test this, the team built VIEWSUITE.

Imagine a 3D map of a real house, made of millions of tiny dots (like a digital cloud of dust).
The AI doesn't have a body; it's just a "camera" floating in this cloud. It can move forward, backward, up, down, and spin around.
The goal is to navigate this cloud to match a target photo.

3. The Failed Attempts: Why "Just Try Harder" Didn't Work

The team tried standard methods to teach the AI, similar to how you might train a dog with treats:

Direct Reinforcement Learning: They let the AI wander around. If it got close to the target, it got a "treat" (a reward).
The Result: It didn't work well. The AI kept wandering aimlessly. Because the "treat" was so rare (finding the exact spot is hard), the AI never learned the right path. It was like trying to teach someone to find a needle in a haystack by only giving them a cookie if they held the needle; they'd never get the cookie, so they'd never learn.

4. The Breakthrough: The "Scrapbook" Strategy

The team realized something clever: Even when the AI fails, it learns something.

If the AI tries to go from Point A to Point B and misses, it still learned that "Point A is connected to Point B."
They realized that every single attempt, successful or not, is a piece of a puzzle.

They created a system called View Graph Distillation. Think of it like this:

Self-Exploration: The AI wanders around the digital house, taking thousands of paths.
The Scrapbook (View Graph): They take all these wandering paths and paste them into a giant "scrapbook" (a graph). This scrapbook maps out exactly how every room connects to every other room.
Distillation (Teaching from the Scrapbook): Instead of waiting for the AI to get lucky, the team takes paths out of this scrapbook and turns them into lessons.
- Old Way: "Go find the target." (Too hard, AI gets lost).
- New Way: "Here is a path from the scrapbook that worked. Here is the start, here is the end, and here are the steps in between. Now, you try to do this."

By constantly switching between wandering (exploration) and studying the scrapbook (distillation), the AI learned to plan.

5. The Result: From Lost to Master

The results were dramatic:

Before: The AI (using a model called Qwen2.5-VL-7B) succeeded only 2.5% of the time. It was basically guessing.
After: With their new "Scrapbook" training method, success jumped to 47.8%.
Comparison: This trained AI became better at this specific task than the most advanced commercial models available (like GPT-5.4 Pro and Gemini 3.1 Pro), which only reached about 18–21%.

The Big Takeaway

The paper proves that AI models can learn to actively plan in 3D space, not just react to what they see. The secret wasn't just giving them more data; it was teaching them to look at their own mistakes as valuable lessons. By turning every failed attempt into a structured map (the View Graph), they helped the AI build a mental map of the world, allowing it to navigate complex spaces with a clear plan.

In short: They taught the AI to stop wandering blindly and start reading its own "map of mistakes" to find the way home.

Technical Summary: VIEWSUITE

Problem Formulation

The paper addresses view planning, defined as the ability of an agent to actively choose a sequence of viewpoint-altering actions to locate a target view within a 3D environment. Unlike prior work limited to 2D image regions, panoramic rotation, or physical embodied navigation (which relies on affordances and traversability), this work focuses on pure 6-DoF viewpoint control in real 3D scenes. The core challenge is decomposing view planning into two coupled abilities:

View-Action Understanding: Predicting how a single action transforms the current view.
Multi-Turn Composition: Leveraging that understanding to compose a sequence of actions over multiple turns to reach a specific target view.

The authors identify a critical planning gap in frontier Vision-Language Models (VLMs): while models often possess basic view-action knowledge (performing well on single-turn tasks), they fail to compose these transformations into effective multi-turn plans, particularly as the distance between viewpoints increases.

Methodology

1. VIEWSUITE Benchmark

To probe these capabilities, the authors introduce VIEWSUITE, a benchmark constructed on approximately 300 real indoor scenes from ScanNet. The environment uses a 3D point-cloud backend rendered via Open3D, supporting 12 discrete 6-DoF actions (6 translations and 6 rotations with fixed step sizes: 0.5m and 30°).

The benchmark comprises three diagnostic tasks:

Path-to-View (P2V): Given an initial view and an action sequence, predict the resulting view (multiple-choice).
View-to-Path (V2P): Given an initial and target view, infer the action sequence that connects them (multiple-choice).
Interactive View Planning (IVP): Given an initial view, a target view, and a top-down reference, the agent must interactively plan a sequence of actions over a fixed turn budget (up to 10 turns) and submit a 6-DoF estimate of the target viewpoint. Success is defined by the distance between the estimated and ground-truth viewpoints falling within calibrated thresholds ( $\beta_t=1, \beta_r=1$ ).

2. Iterative Training Framework

The paper proposes an iterative framework to close the planning gap, alternating between Self-Exploration and View Graph Distillation. This approach is designed to overcome the sparsity of rewards in pure Reinforcement Learning (RL), where direct PPO or GRPO training yields negligible improvements (plateauing at ~3-5% success).

Self-Exploration Stage: The agent interacts with the VIEWSUITE environment using PPO. Crucially, the framework does not discard failed trajectories. Instead, a background process continuously compresses all completed trajectories (successful or not) into a View Graph. In this graph, nodes represent viewpoints and edges represent the actions taken between them.
View Graph Distillation Stage: Paths are sampled from the accumulated view graph and reformulated into supervised training data. The key insight is that any path from node $A$ to node $B$ constitutes a valid view-planning demonstration, regardless of whether the original episode achieved its specific goal. The reformulation operator $\mathcal{R}(P)$ treats the end of the path as the target and the start as the initial view, generating:
1. Multi-turn view planning demonstrations.
2. View difference estimation tasks (predicting unified view distance).
3. View difference multiple-choice questions.
The policy is fine-tuned via Supervised Fine-Tuning (SFT) on this diverse dataset before initializing the next self-exploration iteration.

Key Results

Evaluation of Frontier VLMs

The authors evaluated 13 frontier VLMs (7 proprietary, 6 open-weight) on VIEWSUITE.

Single-Turn Performance: Models achieved ~50–70% accuracy on short-horizon P2V and V2P tasks, indicating non-trivial understanding of local view transitions.
Multi-Turn Failure: Performance collapsed on the Interactive View Planning (IVP) task. The best model, Gemini 3.1 Pro, achieved only 21.4% success, while most open-weight models scored below 5%.
Bottleneck Analysis: Increasing the turn budget or using higher-fidelity rendering (3D Gaussian Splatting) yielded only marginal gains, confirming that the bottleneck is the model's ability to compose view changes, not visual fidelity or exploration horizon.

Performance of the Proposed Framework

Applying the iterative self-exploration and view graph distillation framework to Qwen2.5-VL-7B resulted in significant improvements:

Success Rate: The model's IVP success rate increased from a baseline of 2.5% to 47.8%.
Comparison: This performance surpasses all evaluated frontier models, including GPT-5.4 Pro (18.5%) and Gemini 3.1 Pro (21.4%).
Ablation Studies:
- Direct PPO and GRPO baselines failed to improve beyond ~6%.
- "Success-Only Bootstrapping" (using only successful RL trajectories for SFT) reached only 6.2%, highlighting the importance of distilling knowledge from all trajectories via the view graph.
- Random-graph construction (using random actions) achieved only 13.0%, proving that on-policy graph construction is critical.
Transferability: The spatial priors learned through IVP transferred to related tasks (P2V/V2P) and an external benchmark (MindCube), improving performance by 8–12 points over the base model.

Significance and Claims

The paper claims that interactive view planning is a distinct capability that requires more than just static spatial reasoning or single-step prediction. The primary contribution is demonstrating that VLMs can learn to actively reason and plan in 3D space by leveraging their own exploration data, even when that data is initially sparse and mostly unsuccessful.

The authors assert that their iterative framework effectively bridges the gap between local view-action understanding and global multi-turn planning by:

Transforming sparse RL rewards into dense supervision signals via view graph distillation.
Utilizing hindsight (relabeling failed trajectories as valid transitions) to reshape the policy distribution.
Generating transferable spatial priors that enhance view understanding beyond the specific training task.

The work concludes that while current frontier VLMs struggle with multi-turn 3D planning, a framework combining self-exploration with structured knowledge distillation can enable smaller models (7B parameters) to outperform much larger proprietary models in this domain. The authors note limitations regarding the current focus on static indoor scenes and discrete action spaces, suggesting extensions to dynamic environments and continuous control as future work.