🤖 AI

PanoWorld: Towards Spatial Supersensing in 360 $^\circ$ Panorama World

This paper introduces PanoWorld, a novel framework that enables multimodal large language models to perform robust 360-degree spatial reasoning by treating equirectangular panoramas as continuous, observer-centered spaces through dedicated spherical geometry adaptation and a new diagnostic benchmark.

Original authors: Changpeng Wang, Xin Lin, Junhan Liu, Yuheng Liu, Zhen Wang, Donglian Qi, Yunfeng Yan, Xi Chen

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Changpeng Wang, Xin Lin, Junhan Liu, Yuheng Liu, Zhen Wang, Donglian Qi, Yunfeng Yan, Xi Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are standing in the middle of a room, and you have a camera that can take a picture of everything around you at once—360 degrees, from the floor to the ceiling, with no blind spots. This is a 360-degree panorama.

For a long time, the smartest AI vision models (called MLLMs) have been trained to look at the world like a human with a narrow field of view: they see one flat picture at a time, like looking through a window. If you want them to understand a whole room, you have to show them a series of small, disconnected snapshots and ask them to guess how the pieces fit together. It's like trying to understand a whole puzzle by looking at one piece at a time and hoping you remember where the others were.

PanoWorld is a new system that teaches AI to stop looking through "windows" and start seeing the entire sphere at once. Here is how they did it, broken down simply:

1. The Problem: The "Flat Map" vs. The "Globe"

The paper argues that current AI treats a 360-degree image like a flat, stretched-out map (like a world map that distorts the poles). But the real world is a sphere.

The Old Way: The AI sees a distorted image where the top and bottom are squished and the left and right edges are far apart. It doesn't realize that the left edge and the right edge are actually touching in the real world.
The PanoWorld Way: The AI learns to treat the image as a continuous, observer-centered sphere. It understands that if you turn your head 180 degrees, the object on your left is now on your right, and the "seam" where the image wraps around is just a line, not a wall.

2. The Solution: Teaching the AI "Spherical Intuition"

To fix this, the researchers built a new training system with three main parts:

A. The "Super-Teacher" Data Pipeline

They couldn't just feed the AI random pictures. They needed a way to teach it the rules of 3D space.

The Analogy: Imagine trying to teach a child geography. You can't just show them a flat map; you need a globe.
What they did: They built a massive pipeline that took 570,000 real-world 360-degree photos. They used other smart tools to find objects (like chairs or people), measure how far away they are, and label their exact position on the "sphere" (using angles like "30 degrees to the right and 10 degrees up"). They then verified this data twice to make sure the labels were correct. This created a "gold standard" textbook for the AI.

B. The "Spherical GPS" in the Brain

They modified the AI's architecture (the "brain" of the model) to include a special module called Spherical Spatial Cross-Attention.

The Analogy: Think of a standard AI as a person reading a book on a flat table. PanoWorld adds a 360-degree GPS inside the reader's head.
How it works: Every time the AI looks at a pixel in the image, this GPS tells it: "This pixel isn't just at coordinates (x, y); it is actually pointing in a specific direction in 3D space." This allows the AI to understand that an object on the far left of the image is physically close to an object on the far right, even though they look far apart on the screen.

C. The "Four Superpowers"

The paper defines four specific skills the AI needed to learn to master this:

Semantic Anchoring: Knowing what things are (e.g., "That is a red fire hydrant").
Spherical Grounding: Knowing where they are on the sphere (e.g., "It is 45 degrees to my right").
Reference-Frame Transformation: Understanding how things move if you turn around (e.g., "If I turn left, the fire hydrant is now behind me").
Depth-Aware Reasoning: Understanding how far away things are and their 3D relationship (e.g., "The car is behind the tree").

3. The Results: Why It Matters

The researchers tested their new model, PanoWorld, against the best existing AI models (including big names like GPT-4o and Qwen).

The Test: They used a new benchmark called PanoSpace-Bench designed specifically to test if an AI understands the "wrap-around" nature of 360-degree views.
The Outcome: PanoWorld crushed the competition. While other models struggled to figure out where objects were relative to each other in a full circle, PanoWorld got it right most of the time.
Real-World Transfer: They also tested it on tasks like robot navigation and finding specific people or objects in a room. Even though they didn't train the AI specifically for those tasks, it performed better than models that had been trained for years on them.

The Bottom Line

The paper claims that to truly understand a 360-degree world, you can't just stitch together flat pictures. You have to teach the AI to think in spheres. By giving the AI a "native" understanding of panoramic geometry, they created a system that can reason about space, distance, and direction in a way that feels much more natural and human-like for immersive environments.

In short: They stopped treating 360-degree photos like distorted flat maps and started treating them like the 3D globes they actually are, resulting in an AI that can "see" the whole room at once without getting confused.

Technical Summary: PanoWorld: Towards Spatial Supersensing in 360° Panorama World

Problem Statement

Multimodal Large Language Models (MLLMs) currently struggle with robust spatial reasoning because they are predominantly trained on perspective images, inheriting a narrow field of view (FoV) that limits their ability to perceive the full surrounding environment. While 360° panoramic sensing offers "supersensing" by capturing the entire observer-centered environment, existing MLLM pipelines typically decompose these panoramas into multiple discrete perspective views. This approach treats the spherical structure of equirectangular projection (ERP) as implicit, failing to leverage the globally consistent, wrap-around continuity and viewpoint-dependent spatial relations inherent in panoramic data.

Directly transferring perspective-based models to panoramas is challenging due to fundamental representation gaps, including geometric distortion, non-uniform spatial sampling, and boundary discontinuities. Furthermore, current panoramic MLLM research is fragmented, lacking a systematic capability taxonomy and a unified benchmark to evaluate spatial reasoning beyond standard Visual Question Answering (VQA) metrics.

Methodology

The authors propose PanoWorld, a unified framework for "pano-native" spatial learning that enables MLLMs to reason directly over continuous, observer-centered ERP panoramas. The methodology consists of three core components:

1. Capability Taxonomy

The paper defines pano-native understanding through four capability families, which serve as the foundation for supervision and benchmarking:

Semantic Anchoring: Grounding language to visual entities, attributes, and global scene topology.
Spherical Grounding: Localizing entities on the viewing sphere using yaw and pitch parameters rather than planar image coordinates.
Reference-Frame Transformation: Reasoning about spatial relations under observer rotation (camera rotation) or object-conditioned reorientation, including seam-aware wrap-around continuity.
Depth-Aware 3D Spatial Reasoning: Connecting spherical observations to 3D structure, including depth, relative distance, and viewer-centered relations (e.g., left/right, front/behind).

2. Large-Scale Metadata Construction Pipeline

To support this taxonomy, the authors constructed a dataset of 570K high-quality ERP panoramas from mixed sources (indoor/outdoor, synthetic/real). They developed a verifiable metadata pipeline to generate geometry-aware, language-grounded, and depth-aware supervision:

Geometry-Aware Detection: Panoramas are projected into overlapping perspective views for detection, then reprojected to ERP coordinates with geometric verification (IoU-based suppression, cross-view consistency) to handle distortion and seam splitting.
Language-Grounded Semantics: An MLLM generates semantic annotations and referring phrases, which are verified via a "description-re-detection" loop to ensure alignment between text and visual regions.
Depth Association: Entities are associated with depth information from source data or estimated via panoramic depth models.
Structured Graph: These elements form a metadata graph $G=(V, E)$ representing entities and their spherical/3D relations, from which 2.998M instruction-tuning examples are derived.

3. Pano-Aware Model Adaptation (PanoWorld)

The model is built upon Qwen3.5-VL and extended with a Spherical Spatial Cross-Attention (SSCA) module.

Spherical Token Construction: For each visual patch, the model computes its center in ERP coordinates and maps it to a spherical direction $(\lambda, \phi)$ . These directions are encoded via sinusoidal spherical encoding and projected into the visual hidden space to create spherical tokens $S$ .
Cross-Attention Injection: Immediately after patch embedding, visual tokens $H^{(0)}$ query the spherical tokens $S$ via cross-attention. This allows the model to retrieve geometry-aware signals based on observer-centered directions.
Gated Fusion: The resulting geometry-aware signal is fused back into the visual stream via a gated residual update, preserving the pretrained backbone while injecting spherical geometry.

Key Contributions

Capability Taxonomy: A systematic formulation of panoramic spatial reasoning decomposed into semantic anchoring, spherical grounding, reference-frame transformation, and depth-aware 3D reasoning.
Data & Benchmark: A scalable metadata-driven pipeline producing 570K panoramas with verified graphs, and PanoSpace-Bench, a diagnostic benchmark evaluating ERP-native spatial reasoning across localization, relational reasoning, 3D spatial understanding, and seam continuity.
Model Architecture: The PanoWorld model with Spherical Spatial Cross-Attention, which injects spherical geometry into the visual stream for geometry-consistent understanding.
Empirical Validation: Extensive experiments demonstrating that dedicated pano-native supervision and geometry-aware adaptation are necessary for robust panoramic reasoning.

Experimental Results

The proposed framework was evaluated on PanoSpace-Bench, H*Bench (Human-centric Visual Search), and R2R-CE Val-Unseen (Vision-and-Language Navigation).

PanoSpace-Bench: PanoWorld achieved an overall accuracy of 56.5%, significantly outperforming proprietary models (e.g., GPT-4o at 31.8%) and open-source baselines. Notable gains were observed in BFOV localization (73.3% mIoU vs. <5% for baselines) and seam continuity reasoning (65.5% vs. ~37%).
H*Bench Transfer: In a zero-shot setting, PanoWorld achieved 56.1% overall accuracy, substantially surpassing the best perspective-view baseline (38.4%). After fine-tuning, it reached 70.1%, demonstrating that the learned representation transfers effectively to downstream search tasks without overfitting to the training benchmark.
R2R-CE Val-Unseen: Using only ERP panoramas as input, PanoWorld achieved a Success Rate (SR) of 54.3% and SPL of 52.1%, outperforming recent RGB-only VLN models and waypoint-predictor methods, despite using only 80% of the standard training data.
Efficiency: Direct ERP inference requires a single forward pass, whereas perspective-view rotation methods require 3.5–6.3 sequential interaction steps, making PanoWorld significantly more efficient in terms of interaction steps and token usage.

Significance and Claims

The paper argues that robust panoramic reasoning cannot be achieved by simply treating ERP images as wide 2D images or relying on prompt-level coordinate descriptions. Instead, it requires dedicated pano-native supervision and geometry-aware model adaptation.

The authors claim that their work demonstrates:

Necessity of Pano-Native Learning: General-purpose MLLMs fail at spherical reasoning; specific architectural injection of spherical geometry and training on capability-aligned data are essential.
Unified Representation: Direct reasoning over continuous panoramic space is more efficient and effective than decomposing scenes into fragmented local views.
Transferability: The spatial priors learned from pano-native supervision generalize to diverse downstream tasks, including human-centric visual search and embodied navigation, outperforming methods that rely on perspective-view exploration.

The paper concludes that PanoWorld and PanoSpace-Bench provide a foundation for advancing full-surround spatial intelligence, moving beyond the limitations of the perspective-image paradigm.