💻 computer science

OneCanvas: 3D Scene Understanding via Panoramic Reprojection

OneCanvas introduces a novel 3D scene understanding framework that aggregates multi-view patch features onto a single panoramic canvas with 3D position embeddings, enabling state-of-the-art spatial reasoning with significantly reduced training compute and supporting both situated reasoning and procedural spatial pretraining.

Original authors: Bartłomiej Baranowski, Dave Zhenyu Chen, Matthias Nießner

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bartłomiej Baranowski, Dave Zhenyu Chen, Matthias Nießner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand the 3D world around it, not just by looking at flat pictures, but by understanding where things are, how far apart they are, and what you can see from a specific spot.

Most current AI models try to do this by either building a complex, custom "3D brain" from scratch (which is hard and expensive) or by feeding them millions of examples of 3D questions and answers (which takes a huge amount of computing power).

OneCanvas takes a different, simpler approach. Think of it as turning a messy, multi-angle video of a room into a single, perfect 360-degree panoramic map that the AI can read like a normal picture.

Here is how it works, broken down into simple steps:

1. The "Magic Map" (Panoramic Reprojection)

Imagine you are standing in a room with a 360-degree camera. You take a video, walking around and looking at different objects.

The Old Way: The AI tries to stitch these separate video frames together in its head, guessing where objects are relative to each other. It's like trying to solve a puzzle while wearing blindfolds.
The OneCanvas Way: The system takes every tiny piece of the video (every "patch" of the image), looks at how deep it is, and mathematically "lifts" it out of the flat screen into 3D space.
The Canvas: It then paints all these lifted pieces onto a single, giant, round "canvas" (like a world map). If a chair is to your left, it gets painted on the left side of the map. If a table is far away, it gets painted further out.
The Result: The AI doesn't have to guess anymore. It just looks at this single, giant map. It sees the whole room in one image, with every object in its correct 3D spot.

2. Adding the "Ruler" (3D Position Embedding)

There's a problem with flat maps: they can tell you direction (left/right), but they often lose the sense of distance (how many meters away).

The Fix: OneCanvas adds a special "ruler" to every piece of the map. It attaches a digital tag to every object that says exactly how far away it is in real-world meters.
Why it matters: This lets the AI answer questions like "How big is this room?" or "How far is the door?" without needing to guess. It's like giving the AI a tape measure built right into its eyes.

3. The "Empty Room" Training (Spatial Pretraining)

Before the AI tries to understand real, messy rooms, the researchers teach it in a "virtual sandbox."

The Analogy: Imagine a teacher putting a few toy blocks on a completely empty white table. They ask the student, "How far is the red block from the blue one?"
The Trick: Because the table is empty, the student cannot cheat by memorizing that "red blocks are usually near blue blocks." They must use the ruler and the map to figure it out.
The Benefit: This forces the AI to learn the actual rules of geometry and space, rather than just guessing based on patterns it saw in previous videos. Once it masters this "empty room" logic, it can easily apply it to real, cluttered rooms.

Why is this a big deal?

It's Cheap: Because the AI doesn't need a complex new brain or millions of hours of training, it uses about 10 times less computing power than the best competing methods.
It's Accurate: It currently holds the top spot on major tests for 3D understanding (like SQA3D and VSI-Bench), beating models that are much more complex.
It's Flexible: You can center this "map" on any viewpoint you want. If you want the AI to answer from the perspective of a robot standing in the corner, you just rotate the map. If you want it from the robot's eye level, you rotate it again. The AI handles it all naturally.

In short: OneCanvas turns a confusing 3D video into a single, easy-to-read 360-degree map with built-in rulers. It teaches the AI to understand space by practicing on simple, empty setups first, allowing it to become a 3D expert without needing a supercomputer.

Technical Summary: OneCanvas: 3D Scene Understanding via Panoramic Reprojection

Problem Statement

Existing Vision-Language Models (VLMs) struggle with 3D scene understanding, particularly in situated reasoning tasks (e.g., "How far is the chair from the door?" or "What is visible from this viewpoint?"). Current approaches typically fall into two categories, both of which incur significant costs:

Architectural Complexity: Augmenting pretrained 2D VLMs with dedicated geometry encoders (e.g., point-cloud tokenizers, depth-conditioned encoders) or complex fusion modules. These require joint training at scale to align new geometric components with the language backbone.
Data Scaling: Relying on massive, curated datasets of spatial question-answer pairs to teach geometric reasoning through sheer exposure.

Recent audits suggest that even with these adaptations, many models fail to genuinely reason over geometric inputs, instead relying on statistical regularities of question templates and scene classes. Furthermore, these methods often process scenes piecewise across frames without a unified spatial coordinate system, making situated reasoning from specific viewpoints difficult.

Methodology: OneCanvas

OneCanvas reframes 3D scene understanding as 2D spatial reasoning on a single panoramic canvas. It avoids modifying the VLM's architecture or adding auxiliary loss terms. Instead, it transforms multi-view RGB-D data into a representation that a pretrained VLM can ingest as a standard image.

1. Feature Extraction and 3D Lifting

Given a set of $K$ posed RGB images $\{I_k\}$ with metric depth maps $\{D_k\}$ and camera intrinsics:

A frozen vision encoder (Qwen3-VL) extracts patch-level features $F_k$ from each view.
Each patch $(u, v)$ is "lifted" to a 3D world coordinate $p_{world}$ using its depth $z$ , camera intrinsics, and the camera-to-world pose $T_k$ .
This results in a set of lifted patches $P = \{(p_i, f_i, t_i)\}$ , where $p_i$ is the 3D world coordinate, $f_i$ is the feature, and $t_i$ is the temporal index of the source frame.

2. Panoramic Reprojection

The lifted patches are aggregated onto a single equirectangular panoramic canvas defined by a chosen origin $c$ and orientation $R$ .

Continuous Positioning: Unlike rasterization methods that average or select overlapping patches, OneCanvas places each patch at its continuous angular position (longitude $\theta$ and latitude $\phi$ ) on the canvas. Co-located patches remain distinct tokens.
Position Encoding: The VLM's native 3D Rotary Position Embedding (3D-RoPE) is used, where the spatial dimensions correspond to the continuous longitude and latitude, and the temporal dimension corresponds to the source frame index.
3D Position Embedding: To restore the metric depth lost when collapsing 3D coordinates to angular canvas positions, a 3D position embedding is added to each patch's feature. This embedding encodes the metric offset $(q_x, q_y, q_z)$ in the canvas frame using sinusoidal encodings for axes, radial primitives, and unit-ray directions. This allows the VLM to access metric scale information through its standard attention mechanism.

3. Two-Stage Training

The method employs a two-stage training pipeline to overcome the difficulty of learning geometry from downstream QA alone:

Stage 1: Spatial Pretraining Curriculum: The model is trained on a procedurally generated curriculum where object features (drawn from real images) are placed on an otherwise empty canvas at specific 3D coordinates. Tasks include metric measurement, egocentric direction, navigation, observability, counting, and bounding-box readout. This stage forces the model to learn geometric reasoning without relying on scene-specific priors or class identity shortcuts.
Stage 2: Target Adaptation: The weights from Stage 1 are merged into the base model. A new, lower-rank LoRA adapter is trained on downstream spatial QA datasets (SQA3D, VSI-Bench, ViCA) to adapt the geometric reasoning skills to real-world scene distributions.

Key Contributions

Panoramic Feature Reprojection: A mechanism that aggregates patch features from all views onto a single equirectangular canvas. This matches the input format of pretrained VLMs, requires no architectural modifications, and supports flexible canvas origins for situated reasoning.
Spatial Pretraining Curriculum: A novel training strategy enabled by the panoramic representation. By procedurally placing patch features on an empty canvas, the authors generate on-the-fly supervision for diverse spatial tasks with controlled answer distributions, effectively suppressing dataset-statistic shortcuts.

Results

OneCanvas achieves state-of-the-art performance on major 3D spatial reasoning benchmarks while using significantly less training compute than competing methods:

SQA3D: Achieves 65.3 EM@1 (2.3 points above the previous best), with significant gains in "Which" and "Can" question types that test situated reasoning.
VSI-Bench: Achieves 70.1 average accuracy, leading on route planning and room size estimation.
SPBench: Achieves 72.1 zero-shot overall accuracy (4.8 points above the next best), demonstrating strong generalization to out-of-distribution data.
Efficiency: The method uses an order of magnitude less training compute (approx. 290 A100-equivalent GPU-hours) compared to the strongest competing methods (e.g., SenseNova-SI at ~27,648 A100-equivalent GPU-hours).

Ablation studies confirm that both the panoramic reprojection and the spatial pretraining curriculum are critical, particularly for tasks requiring metric reasoning and multi-step navigation.

Significance and Limitations

The paper claims that OneCanvas demonstrates that high-quality 3D scene understanding can be achieved without complex geometry encoders or massive curated datasets, provided the input representation is structured to align with the VLM's pretrained capabilities. By unifying multi-view data into a single spatial coordinate system and explicitly injecting metric position embeddings, the method enables the VLM to reason about 3D space using its native attention mechanisms.

Limitations:

Dependency on Depth and Pose: The method requires metric depth and camera poses, which pure RGB methods avoid. While feed-forward reconstruction can estimate these, errors can degrade the canvas.
Global Panorama Trade-off: Representing the scene as a single global panorama trades fine spatial precision for compactness, potentially limiting performance on very large or outdoor scenes.
Curriculum Design: The pretraining curriculum is hand-authored; introducing qualitatively new spatial skills requires writing new task generators rather than simply collecting more data.
Backbone Requirements: The method assumes a backbone with at least 2D position encoding (e.g., multi-axis RoPE) to carry longitude and latitude, which may not hold for 1D-position models.