More than the Sum: Panorama-Language Models for Adverse Omni-Scenes

This paper introduces the Panorama-Language Modeling (PLM) paradigm and the PanoVQA dataset to enable holistic $360^\circ$ vision-language reasoning in adverse omni-scenes, demonstrating that a unified panoramic approach yields superior understanding compared to stitching multiple narrow-field-of-view inputs.

The Gist

Here is an explanation of the paper "More than the Sum: Panorama-Language Models for Adverse Omni-Scenes" using simple language and creative analogies.

The Big Idea: Seeing the Whole Picture vs. The Puzzle Pieces

Imagine you are trying to understand a chaotic traffic scene.

The Old Way (Current AI):
Most current AI models are like a person wearing goggles with a tiny, narrow tunnel vision. To see the whole street, they have to take six separate photos (one for each direction), look at them one by one, and then try to mentally stitch them together like a puzzle.

• The Problem: When they try to put the puzzle pieces together, they often lose the "big picture." They might forget that the car on the left is actually connected to the road on the right. They miss the "wrap-around" nature of the world, where the left side of your vision connects seamlessly to the right side. It's like trying to understand a story by reading only one sentence at a time from six different books.

The New Way (This Paper's Solution):
The authors introduce PLM (Panorama-Language Modeling). Instead of looking through a narrow tunnel, this AI wears 360-degree goggles. It sees the entire world in one giant, seamless circle.

• The Analogy: Think of the old AI as a photographer taking six separate snapshots and pasting them on a wall. The new AI is like a fish-eye lens that captures the whole room in a single, fluid image. It understands that the "front" and the "back" are part of the same continuous loop.

---

The Three Key Ingredients

To make this new AI work, the team built three main things:

1. The "Super-Training Manual" (PanoVQA Dataset)

AI needs to learn by doing. The authors created a massive new textbook called PanoVQA.

• What's in it? It has over 650,000 questions and answers about driving scenes.
• Why is it special? Most driving datasets only show "perfect" sunny days. This one is like a chaotic driving simulator. It includes:
  • Normal driving: Just cars and roads.
  • Occlusion: Things hiding behind other things (like a pedestrian hiding behind a bus).
  • Accidents: Crashes and dangerous situations.
• The Metaphor: If old datasets were like a driving school with only empty parking lots, this dataset is like a driving test in a heavy rainstorm during rush hour. It forces the AI to learn how to think, not just how to recognize a red light.

2. The "Smart Brain" (Panorama Sparse Attention)

Processing a 360-degree image is computationally heavy. It's like trying to read a book where every page is glued to the next one in a giant circle. If you try to read every single word at once, your brain (or computer) explodes.

• The Solution: The authors invented a new attention mechanism called PSA (Panorama Sparse Attention).
• The Metaphor: Imagine you are in a crowded stadium.
  • Old AI: Tries to look at every single person in the stadium at the same time. It gets overwhelmed and misses the important stuff.
  • New AI (PSA): It has a smart spotlight. It knows to focus intensely on the players on the field (the cars and pedestrians) and the referee (the road), while ignoring the empty seats in the sky (the clouds) or the blurry background. It dynamically decides what to look at, saving energy while keeping the most important details sharp.

3. The "Plug-and-Play" Upgrade

Usually, to make a new type of AI, you have to rebuild the whole engine from scratch.

• The Innovation: This new "Panorama Sparse Attention" module is like a universal adapter. You can take an existing, powerful AI (like Qwen or LLaVA) that was trained on normal photos, plug this new module in, and suddenly it can understand 360-degree panoramas without needing to be retrained from zero.

---

Why Does This Matter? (The Results)

The team tested their new AI against the best existing models. Here is what happened:

• The "Wrap-Around" Win: In a test where a pedestrian was standing near the edge of the camera view, the old AI (using 6 separate cameras) got confused. It thought the person was in a different direction because it couldn't see the "seam" where the images joined. The new AI, seeing the whole circle, knew exactly where the person was.
• Handling Chaos: When faced with a cluster of bicycles or a potential crash, the new AI made safer, more logical decisions because it saw the entire context, not just fragments.
• The Verdict: The new model didn't just do a little better; it was significantly smarter. It proved that seeing the whole world at once is "more than the sum of its parts."

Summary in One Sentence

This paper teaches AI to stop looking at the world through six tiny windows and start seeing it through one giant, seamless 360-degree lens, allowing it to understand dangerous and complex driving situations much better than ever before.

Technical Summary

Here is a detailed technical summary of the paper "More than the Sum: Panorama-Language Models for Adverse Omni-Scenes".

1. Problem Statement

Existing Vision-Language Models (VLMs) are predominantly designed for "pinhole" imagery (narrow Field of View). When applied to omnidirectional (360°) scenes, such as those in autonomous driving, robotics, and AR/VR, current methods rely on a "stitching" approach: sampling multiple narrow-view crops, processing them individually, and attempting to reconstruct a holistic understanding.

This approach suffers from two critical limitations:

1. Loss of Spatial Continuity: Multi-view perception breaks the seamless 360° continuity, failing to model the "wrap-around" nature of panoramas where the left and right edges are connected. This leads to spatial ambiguities and errors in reasoning about object relationships across camera boundaries.
2. Lack of Benchmarks and Architecture Compatibility: There is a scarcity of large-scale benchmarks for panoramic VQA, particularly those covering adverse scenarios (occlusions, accidents). Furthermore, standard Transformer-based VLMs use dense self-attention with $O(L^2)$ complexity, which is computationally prohibitive for high-resolution equirectangular projections (ERP) and fails to account for geometric distortions inherent in panoramic images.

2. Methodology

The authors propose Panorama-Language Modeling (PLM), a unified framework for 360° vision-language reasoning that treats the panorama as a single, holistic input rather than a collection of views.

A. PanoVQA Benchmark

To address the data gap, the authors constructed PanoVQA, the first large-scale panoramic VQA dataset containing 653,000 QA pairs.

• Data Sources: Aggregated and synthesized from NuScenes (normal driving), BlendPASS (occlusion), and DeepAccident (accidents).
• Scenario Diversity: The dataset covers three distinct categories:
  • Normal (N): General driving, object counting, and spatial relationships.
  • Occluded (O): Complex occlusion reasoning, inferring actions of hidden objects.
  • Accident (D): Risk analysis, collision severity, and avoidance planning.
• Generation Pipeline: Uses a geometry-based stitching pipeline to convert multi-view data into equirectangular panoramas. A structured quadruple representation (category, direction, distance, visibility/speed) is used to annotate objects, which then feeds into an LLM (GPT-5-mini) to generate diverse QA pairs.
• Scale: 538K training samples and 115K validation samples.

B. Panorama-Language Model (PLM) Architecture

The core innovation is the Panoramic Sparse Attention (PSA) mechanism, designed to be a plug-and-play module for existing pre-trained VLMs (e.g., Qwen2.5-VL).

1. Panoramic Hybrid Attention (PHA): The model replaces standard dense attention with a parallel combination of:
   • Sliding Window Attention (SWA): Captures fine-grained local patterns within non-overlapping windows, reducing complexity from $O(L^2)$ to $O(L \cdot L_w)$.
   • Panoramic Sparse Attention (PSA): A global head that dynamically selects the Top-K most relevant key tokens for each query token.
     • Dynamic Selector: Uses a lightweight gating network with learnable position embeddings to compute relevance scores between tokens, filtering out uninformative regions (e.g., sky) and focusing on semantic horizons.
     • Wrap-Around Handling: Unlike fixed sparse attention, PSA learns to connect distant tokens (e.g., front-left to back-right) to preserve the 360° topology.
2. Efficiency: This hybrid approach allows the model to process high-resolution panoramas efficiently while maintaining compatibility with pre-trained weights, avoiding the need for training from scratch.

3. Key Contributions

1. PanoVQA Dataset: A comprehensive benchmark with 653K QA pairs covering normal, occluded, and accident scenarios, filling a critical gap in panoramic VQA evaluation.
2. Panorama-Language Modeling (PLM): A new paradigm demonstrating that holistic 360° understanding yields performance "greater than the sum" of multi-view stitching.
3. Panoramic Sparse Attention (PSA): A novel, parameter-efficient attention module that handles equirectangular distortions and long-range dependencies dynamically, enabling existing VLMs to process panoramas without full retraining.
4. Empirical Validation: Extensive experiments proving that PLM outperforms state-of-the-art VLMs and multi-view baselines, particularly in spatial reasoning and safety-critical tasks.

4. Experimental Results

Performance on PanoVQA

• Superiority over VLMs: The proposed PLM (based on Qwen2.5-VL-7B) achieves a normalized score of 45.48%, significantly outperforming the best zero-shot open-source model (InternVL3-8B-Ins at 34.48%) and proprietary models like Gemini-2.5-Flash-Lite (30.74%).
• Multi-view vs. Panorama: In a controlled comparison, the 1-Pano model (SFT) achieved 41.42% accuracy, outperforming the 6-Cam multi-view baseline (40.22%).
  • Qualitative Insight: The panoramic model correctly identified spatial directions (e.g., "front") and visibility, whereas the multi-view model hallucinated directions (e.g., "front-left") due to fragmented context.
• Robustness: PLM showed the most significant gains in Occlusion (O) and Accident (D) scenarios, where holistic context is crucial for reasoning about hidden objects and collision risks.

Ablation Studies

• PSA Effectiveness: Removing the dynamic gating mechanism (using Simplified Sparse Attention) resulted in lower performance (32.14% vs. 41.49% with full PSA), confirming the importance of dynamic token selection.
• Parameter Efficiency: The PSA module allows the model to achieve near-full fine-tuning performance with significantly fewer trainable parameters (95M vs. 668M for the vision components alone).
• Scaling Laws: Performance scales smoothly with data volume, validating the use of the PanoVQA-mini subset for rapid iteration.

5. Significance and Conclusion

This work establishes that holistic panoramic understanding is fundamentally different from and superior to multi-view stitching for tasks requiring spatial coherence.

• Safety Criticality: By enabling models to reason about the entire 360° environment simultaneously, PLM improves the reliability of autonomous driving agents in detecting occluded hazards and predicting accident severity.
• Architectural Shift: The introduction of PSA provides a scalable solution for adapting large language models to non-standard, high-resolution, and geometrically distorted inputs (like equirectangular projections) without sacrificing efficiency.
• Future Impact: The paper sets a new standard for evaluating vision-language systems in omnidirectional environments, moving the field beyond simple object recognition toward complex, context-aware reasoning in adverse real-world scenes.

The code and dataset are available at the project page: https://github.com/InSAI-Lab/PanoVQA.