💻 computer science

Beyond Thinking: Imagining in 360 $^\circ$ for Humanoid Visual Search

This paper proposes "Imagining in 360°," a novel framework that decouples Humanoid Visual Search into a probabilistic Imaginator and an Actor to infer semantic spatial priors in a single step, thereby eliminating the need for expensive trajectory-level annotations while significantly improving search efficiency and success rates in complex 360° environments.

Original authors: Jingdong Zhang, Yizhou Wang, Zhengzhong Tu, Xin Li, Wenping Wang, Xiaohang Zhan

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jingdong Zhang, Yizhou Wang, Zhengzhong Tu, Xin Li, Wenping Wang, Xiaohang Zhan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific item, like a lost set of keys, in a giant, dark warehouse. You can only see a small circle of light in front of you.

The Old Way: The "Overthinker"
Previous methods tried to solve this by giving the robot a single brain that had to do everything at once. It had to look at the small circle of light, remember everything it saw, guess what might be in the dark corners, and then decide where to turn next.

The Problem: This is like asking a person to solve a complex math problem while simultaneously trying to navigate a maze. It gets confused, makes slow guesses, and often spins in circles because it's too busy "thinking" to "imagining" the whole room. It also requires a human teacher to write out every single step of the robot's thought process for every possible scenario, which is incredibly expensive and slow to create.

The New Way: "Imagining in 360°"
This paper proposes a smarter team approach. Instead of one overworked brain, they split the job into two specialized roles: The Imaginator and The Actor.

1. The Imaginator (The "Mental Map Maker")

Think of the Imaginator as a psychic cartographer. Its only job is to stand in the center of the room and say, "Okay, I see a door here. Based on how rooms usually work, there's probably a hallway to the left and a staircase behind me, even though I can't see them yet."

How it works: Instead of guessing one specific spot, it creates a list of possibilities. It says, "There's a 60% chance the keys are near the stairs, a 30% chance they're by the door, and a 10% chance they're on the shelf."
The Magic: It doesn't need to see the whole room to make these guesses. It uses "common sense" about how spaces are built (e.g., stairs usually lead to floors, doors lead to other rooms). It generates these guesses instantly and cheaply, without needing a human to teach it every single step.

2. The Actor (The "Explorer")

The Actor is the robot's body and eyes. It receives the list of guesses from the Imaginator.

The Process: Instead of wandering blindly, the Actor looks at the Imaginator's list. "Hmm, the Imaginator thinks the keys are likely near the stairs. Let's turn there first."
The Result: The Actor moves efficiently, checking the most likely spots first. If it sees something that contradicts the guess (e.g., no stairs, just a wall), it updates its plan and checks the next item on the list.

Why This is a Big Deal

Speed and Efficiency: By separating the "guessing" from the "moving," the robot stops spinning in circles. It goes straight to the most likely spots. In the paper's tests, this method helped robots find objects much faster and more often than before, even in very messy or complex environments.
The "Free" Training Data: The biggest breakthrough is how they taught the Imaginator. Because the Imaginator only needs to guess the layout of a room based on a tiny slice of it, the researchers could use a computer to generate 1.92 million training examples automatically. They didn't need humans to write out millions of stories about how a robot should search. They just let the computer practice guessing layouts over and over.
Working with Any Brain: This system is like a "plug-and-play" upgrade. You can take a standard robot brain (even a smaller, cheaper one) and plug in this "Imaginator" module, and suddenly it becomes much better at searching.

The Bottom Line

The paper argues that to find things in a big, 360-degree world, you shouldn't just "think" harder. You need to imagine the whole space first. By having one part of the system build a mental map of the unseen world and another part act on that map, the robot becomes a much more efficient and confident explorer.

Technical Summary: Imagining in 360° for Humanoid Visual Search

Problem Definition

Humanoid Visual Search (HVS) requires embodied agents to actively explore immersive 360° panoramic environments to locate specific objects (Humanoid Object Search, HOS) or navigable paths (Humanoid Path Search, HPS). Unlike passive perception, HVS demands a closed-loop perception-action cycle where agents must reorient their "head" (camera) to uncover occluded information.

Current state-of-the-art approaches, such as Thinking in 360° (Yu et al., 2025), treat HVS as a monolithic task driven by Multimodal Large Language Models (MLLMs) using cumulative Chain-of-Thought (CoT) reasoning. These methods suffer from three primary bottlenecks:

Cognitive Overload: Coupling complex 3D spatial understanding with action planning in a single model creates significant reasoning burdens for current MLLMs.
Implicit Guessing: Without explicit modeling of spatial relationships, agents are forced to implicitly guess the layout of unseen regions, leading to inactive or inaccurate explorations.
Data Scalability: Training these monolithic agents requires expensive, high-quality, multi-turn CoT trajectory annotations, which are difficult to curate at scale and hinder generalization to diverse, in-the-wild scenarios.

Methodology: A Decoupled Architecture

The paper proposes Imagining in 360°, a framework that decouples the exploration process into two specialized components: a probabilistic Imaginator and an executive Actor.

1. The Imaginator (Probabilistic Spatial Prior Generator)

The Imaginator is designed to explicitly model the 360° environment by predicting the semantic layout of both observed and unobserved regions in a single step, rather than maintaining a cumulative reasoning chain.

Task: Given a limited Narrow Field-of-View (NFoV) observation and a natural language instruction, the Imaginator outputs a structured block containing:
- [Observed]: Landmarks currently or previously visible.
- [Imagined]: Probabilistic predictions of objects likely existing in unseen regions (e.g., "Staircase: (-45, -10)").
Probabilistic Sampling: Instead of a single deterministic guess, the Imaginator generates multiple spatial hypotheses via temperature sampling. This captures the inherent uncertainty of occluded environments.
Output: A ranked list of suggestions (e.g., "Suggest Check (φ, µ)") representing a distribution of spatial priors.

2. The Actor (Execution Agent)

The Actor is responsible for the physical search and can be any off-the-shelf MLLM or fine-tuned search model.

Integration: The Actor receives the current NFoV observation, historical memory, and the spatial suggestions generated by the Imaginator.
Reasoning: Empowered by these priors, the Actor evaluates the plausibility of the suggested layouts against its visual memory to generate a reasoning chain.
Action: The Actor outputs a next action, either a rotation (Rot) to a new coordinate or a submission (Sub) of the final target view.

3. Scalable Data Engine

A key innovation is the ability to generate training data without manual trajectory annotations.

Pseudo-Labeling: The authors utilize a powerful teacher model (Qwen3-VL-235B) to label complete 360° panoramas.
Synthetic Trajectories: From these fully annotated panoramas, they synthetically sample trajectories, revealing only 1–8 NFoV observations per instance while keeping 50% of targets out of view.
Scale: This automated pipeline yields approximately 1.92 million high-quality, single-step spatial reasoning samples across ~20,000 diverse scenes, bypassing the cost of trajectory-level CoT curation.

4. Inference Pipeline

During inference, the Imaginator generates $K$ candidate coordinates per step:

One greedy anchor ( $T=0$ ) for the most confident prediction.
$K-1$ stochastic samples ( $T>0$ ) to capture diverse hypotheses.
These absolute coordinates are converted into relative rotation actions for the Actor. A temperature decay schedule is applied over search steps, transitioning from broad exploration to focused prediction as visual evidence accumulates.

Key Contributions

Decoupled HVS Paradigm: The paper redefines visual search by separating intuitive spatial imagination from action planning. This allows agents to explore more actively and confidently, grounded in sound spatial assumptions rather than exhaustive logical chains alone.
Probabilistic Imaginator: Introduction of a specialized model that learns intuitive spatial priors (e.g., architectural layouts) and guides the Actor with probabilistic estimations, effectively hedging against spatial uncertainty.
Scalable Data Engine: A fully automated pipeline that scales training to ~1.92M samples without manual trajectory labels, enabling robust performance in the wild without the prohibitive cost of expert annotation.

Experimental Results

The framework was evaluated on H*Bench across various Open-Source, Proprietary, and Finetuned MLLMs.

Performance Gains: The decoupled pipeline significantly outperforms monolithic baselines.
- Qwen3-VL (235B): HOS success rate increased from 23.62% to 65.04%; HPS from 21.44% to 38.25%.
- HVS-3B (RL-finetuned baseline): Success rate jumped from 48.04% to 62.75% on HOS when integrated with the Imaginator.
- Proprietary Models: Gemini-2.5 Pro improved from 55.00% to 66.10% on HOS.
Efficiency: The joint pipeline shifts the search distribution, with most searches converging within 1–3 steps, compared to the baseline often hitting the 10-step failure limit. Inference latency is reduced to ~2.5s per step (vs. 20–30s for pixel-level imagination baselines like FLUX).
Ablation Studies:
- Removing the structured layout context drops performance below the baseline, confirming the necessity of explicit global spatial modeling.
- Probabilistic sampling ( $T>0$ ) outperforms deterministic greedy decoding ( $T=0$ ), validating the importance of capturing spatial uncertainty.
- Scaling laws show monotonic improvement as the Imaginator's pretraining data increases from 0 to 1.92M samples.

Significance and Claims

The paper claims that Imagining in 360° fundamentally shifts how embodied agents approach unobserved space. By explicitly modeling spatial priors in language space rather than relying on pixel-level reconstruction or monolithic CoT reasoning, the framework:

Reduces Cognitive Burden: Separates the difficult task of spatial layout prediction from action planning.
Enables Scalability: Solves the data bottleneck by generating millions of training samples automatically, making high-performance HVS accessible without expensive expert annotations.
Improves Robustness: Demonstrates superior performance in complex, in-the-wild environments where agents typically fail due to spatial uncertainty or hallucination.

The authors conclude that while the current framework operates in an open-loop execution paradigm (where the Actor cannot feedback to the Imaginator), it successfully establishes a complementary architecture that grounds generative spatial priors with logical action planning, offering a scalable and efficient path forward for Humanoid Visual Search.