PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning

This paper introduces PanoEnv, a large-scale 3D spatial reasoning benchmark for panoramic images, and proposes a curriculum-based reinforcement learning framework with GRPO that significantly enhances the 3D spatial intelligence of Vision-Language Models, achieving state-of-the-art performance on omnidirectional perception tasks.

The Gist

Imagine you are trying to teach a robot how to navigate a room, but instead of showing it a normal photo, you hand it a 360-degree fisheye panorama.

In a normal photo, a chair looks like a chair. But in a 360-degree "fisheye" image, the room is stretched out like a piece of taffy. The ceiling is at the top, the floor is at the bottom, and the walls are wrapped around the sides. Objects near the top and bottom look squished and distorted.

The Problem:
Current AI models (Vision-Language Models) are like students who have only ever studied flat, normal photos. When you show them a fisheye panorama, they get confused. They might think a tiny, squished object far away is actually a giant object right in front of them. They struggle to answer simple questions like, "Is the lamp to the left of the sofa?" or "How big is that table really?" because the image is lying to them about the shape and distance.

The Solution: PanoEnv
The researchers created a new training ground called PanoEnv. Think of it as a virtual video game world where they can generate perfect, 360-degree photos of rooms.

Here is the magic trick: Because this world is a computer simulation, the researchers know the exact truth. They know the real distance between the lamp and the sofa, the real volume of the table, and exactly which part of the room the camera is looking at. They used this "perfect truth" to create 14,800 questions and answers.

The Training Method: The "Coach" Approach
The researchers didn't just show the AI the answers. They used a technique called Reinforcement Learning, which is like training a dog with treats.

1. The "Ground Truth" Treat: Instead of letting a human or another AI guess if the answer is right (which can be messy), they used the computer's perfect 3D data. If the AI says the table is 2 meters away and the computer says it's actually 2 meters away, the AI gets a "treat" (a reward). If it's wrong, it gets no treat.
2. The Two-Stage Curriculum (The "Gym" Strategy):
   • Stage 1 (The Basics): First, they taught the AI simple, multiple-choice questions (like "Is the door open? Yes/No"). This was like teaching the student to walk before they run. It helped the AI get used to the format without getting overwhelmed.
   • Stage 2 (The Marathon): Once the AI was confident, they introduced the hard stuff: open-ended questions where the AI has to write its own answer (like "Describe the layout of the room"). Because the AI had already learned the basics in Stage 1, it didn't forget how to speak properly while learning the hard stuff.

The Results
Before this training, the best AI models got about 49% of the questions right, and they were terrible at open-ended questions (getting only 6% right).

After the PanoEnv training:

• The AI's overall score jumped to 53%.
• Its ability to answer open-ended questions more than doubled (from 6% to nearly 15%).
• Most impressively, a small AI model (7 billion parameters) trained this way performed better than massive models (32 billion parameters) that hadn't been trained on this specific 3D logic.

The Big Picture
This paper proves that if you give an AI a way to learn the real physics of a 3D world (even if that world is a simulation), it can learn to "see" in 3D, even when looking at a distorted 2D image. It's like teaching a person to drive a car by first practicing in a perfect video game simulator, so when they get on the real road, they already understand how the car moves in space.

In short: They built a perfect 3D simulator, taught an AI to play a game inside it using strict rules, and now that AI is much better at understanding 360-degree views than any AI before it.

Technical Summary

1. Problem Statement

The paper addresses the significant gap in 3D spatial reasoning capabilities of current Vision-Language Models (VLMs) when processing 360° Equirectangular Projection (ERP) images. While VLMs excel at standard perspective images, they struggle with panoramic data due to:

• Geometric Distortion: ERP images stretch pixels near the poles, distorting scale and shape cues learned from pinhole cameras.
• Non-linear Spatial Mapping: 3D relationships (e.g., "left of," "above") map non-linearly to 2D ERP coordinates, making reasoning difficult.
• Lack of 3D Supervision: Existing VQA datasets often lack precise physical ground truth (depth, 3D bounding boxes, volume) aligned with omnidirectional views, preventing models from learning true 3D geometry.
• Performance Collapse: State-of-the-art (SOTA) VLMs achieve low accuracy (49%) on panoramic tasks, with a near-total failure on open-ended (OE) questions (6–8%), indicating an inability to generate metric-consistent or complex spatial descriptions.

2. Methodology

The authors propose a comprehensive framework consisting of a new benchmark dataset and a specialized Reinforcement Learning (RL) post-training strategy.

A. PanoEnv-QA Benchmark

A large-scale dataset constructed from TartanAir (a synthetic 3D environment with precise ground truth).

• Scale: 14,827 QA pairs across 595 panoramic scenes.
• Ground Truth: Every question is programmatically generated from accurate 3D annotations (depth maps, semantic segmentation, 3D bounding boxes), ensuring pixel-level alignment between appearance and geometry.
• Five Question Categories:
  1. Camera View Source Identification: Determining which of the 6 cubemap faces an object appears in (testing seam awareness).
  2. Object Distance Estimation: Quantitative depth reasoning (median, percentiles).
  3. Environment Identification: Classifying scene attributes (indoor/outdoor, urban/nature).
  4. Relative Spatial Positioning: Reconstructing 3D Cartesian coordinates to determine relations (e.g., "behind and to the right").
  5. Intrinsic Attribute Comparison: Comparing physical properties independent of view (e.g., true 3D volume, shape flatness).

B. 3D-Aware RL Post-Training Framework

To bridge the performance gap, the authors employ Group Relative Policy Optimization (GRPO) on a Qwen2.5-VL-7B backbone.

• Ground-Truth-Guided Routed Reward: Instead of relying on noisy LLM judges, the reward is calculated directly against the dataset's physical ground truth. A routing mechanism selects one of five specialized strategies based on question type:
  • Yes/No: Strict string matching.
  • Multiple Choice: Subject extraction and normalization.
  • Distance: Numerical parsing with unit conversion and relative error tolerance (e.g., <10% error = full reward).
  • Spatial: Axis-wise matching (front/back, left/right, up/down) with partial credit.
  • Counting: Exact numerical match.
• Two-Stage Curriculum Learning: To prevent catastrophic forgetting and stabilize training:
  • Stage 1 (Structured): Trains exclusively on low-entropy tasks (True/False, Multiple Choice) to establish format adherence and discrete decision-making.
  • Stage 2 (Mixed): Fine-tunes on a balanced mix of structured and Open-Ended questions to recover generative skills and improve complex reasoning without losing structured performance.

3. Key Contributions

1. PanoEnv-QA Dataset: The first large-scale, geometry-grounded VQA benchmark for ERP images, offering 14.8K questions with verifiable 3D physical annotations (depth, volume, 3D boxes).
2. Comprehensive Benchmarking: Evaluation of 14 SOTA VLMs (including InternVL, DeepSeek-VL, LLaVA, Qwen) reveals a critical failure in 3D reasoning, particularly for open-ended generation (best OE accuracy: 8.36%).
3. 3D-Aware RL Framework: A novel GRPO-based training pipeline featuring:
   • A multi-faceted reward system anchored in physical reality rather than semantic similarity.
   • A two-stage curriculum that successfully balances discrete decision-making with free-form generation.
   • Achievement of SOTA performance using a 7B parameter model, outperforming much larger 32B models.

4. Results

The proposed PanoEnv-RL model (7B) demonstrates significant improvements over baselines:

• Overall Accuracy: Increased from 49.34% (Zero-shot 7B) to 52.93%.
• Open-Ended (OE) Accuracy: Dramatic improvement from 6.39% to 14.83% (a 132% relative increase).
• Semantic Quality: Achieved the highest LLM-judge scores (Q-Score: 6.24, P-Score: 5.95), surpassing the 32B baseline (Q-Score 5.02).
• Generalization: The model trained on synthetic data (TartanAir) showed strong Sim-to-Real transfer, outperforming the base 7B and even the 72B model on the real-world OSR-Bench for object counting and relative distance tasks.

5. Significance

• Paradigm Shift: Demonstrates that physical ground truth is superior to semantic heuristics for training 3D spatial reasoning. By anchoring rewards in geometry, the model learns to "understand" 3D space rather than just pattern matching 2D textures.
• Efficiency: Proves that a smaller model (7B) with specialized curriculum-based RL can outperform significantly larger models (32B/72B) on complex spatial tasks, suggesting that reasoning logic is more critical than parameter scale for 3D perception.
• Robustness: The two-stage curriculum effectively mitigates the trade-off between structured task performance and open-ended generative capabilities, a common challenge in RL fine-tuning.
• Future Impact: Provides a foundational framework for developing embodied agents, autonomous driving systems, and VR/AR applications that require holistic, distortion-aware 3D scene understanding.