💻 computer science

Enhancing In-context Panoramic Generation via Geometric-aware Pretraining

The paper introduces Canvas360, a two-stage framework that leverages a newly created 1M-sample dataset and geometry-aware pretraining techniques to achieve state-of-the-art performance in diverse in-context panoramic generation tasks while ensuring superior geometric consistency and global coherence.

Original authors: Haoran Feng, Ruiyang Zhang, Longyi Zhang, Dizhe Zhang, Lu Qi

Published 2026-07-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haoran Feng, Ruiyang Zhang, Longyi Zhang, Dizhe Zhang, Lu Qi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to paint a perfect 360-degree panorama of a room on a flat piece of paper. The problem? When you wrap a flat map around a sphere, the top and bottom get squished and stretched like a weird, melted cheese pizza. Most AI art generators try to fix this by using special "cube" maps or 3D tricks, but the authors of this paper, Canvas360, suggest those methods still leave the geometry looking a bit wonky. They argue that if you want a truly seamless, distortion-free panoramic world, you can't just look at the picture; you need to understand the depth and the shape of the space itself.

So, how did they fix it? They built a two-stage training pipeline that acts like a master architect and then a versatile painter.

Stage 1: The Geometry Boot Camp
First, the team taught their AI model to see the world in 3D, not just 2D. They didn't just show the model pictures; they paired every single image with a depth map (a blueprint showing how far away every object is). They fed 100,000 of these paired samples into the model.

To make sure the model didn't get confused between the picture and the blueprint, they used a clever trick: they gave the depth map a "positional offset," like giving it a different seat number in a theater so it knows it's not the same as the picture next to it. They also added a special rule called a "similarity loss," which basically scolded the model if the picture and the depth map started looking too much alike, forcing them to stay distinct.

The coolest part? They introduced velocity circular padding. Imagine a video game world where if you walk off the right edge of the screen, you instantly appear on the left. The authors made sure the AI understood that the left and right edges of a panorama are actually neighbors on a sphere. They didn't just copy-paste the edges; they taught the model that the "velocity" (the direction of change) at the edge must flow perfectly into the other side. This helped the model learn to keep the seams invisible.

Stage 2: The Versatile Painter
Once the model understood the geometry, they moved to the second stage: teaching it to be a master of in-context generation. This means the AI can now take an existing image and a text prompt to do all sorts of magic tricks without needing the depth maps anymore. They trained it on a massive new dataset called Canvas360Dataset, which contains 1 million high-quality samples.

This dataset was built by synthesizing 900,000 new examples covering four specific tasks:

Style Transfer: Turning a photo into a pencil sketch or a painting (200,000 samples).
Outpainting: Expanding the view beyond the original borders (250,000 samples).
Inpainting: Filling in missing holes in the image (250,000 samples).
Editing: Removing objects (like a sofa) or adding new ones (200,000 samples).

The authors explicitly argue against the idea that you need to train separate models for each of these tasks or rely on older "cube map" methods. Instead, they suggest that a single, unified model, pre-trained on geometry, can handle all these jobs better.

Did it work?
The results suggest a strong improvement. When they tested the model, it scored the best on a specific metric called FAED (which measures how faithful the panorama is to the geometry) and performed very well on other quality checks. In a user study with 71 people looking at 10 images, Canvas360 was the favorite for having the most seamless boundaries and the best overall quality.

The authors note that while previous methods often left blurry edges or distorted objects when editing a panorama, Canvas360 managed to keep the 3D structure intact. They didn't claim to have solved every problem in the universe, but their experiments suggest that by teaching the AI to respect the spherical shape of the world from the very beginning, they created a tool that generates much more consistent and realistic panoramic images.

Technical Summary: Canvas360 – Enhancing In-context Panoramic Generation via Geometric-aware Pretraining

Problem Statement

While text-to-panorama generation has advanced significantly, in-context panoramic generation (editing, inpainting, outpainting, and style transfer conditioned on user-provided images) faces distinct challenges. The dominant equirectangular projection (ERP) representation inherently introduces latitude-dependent distortions, making it difficult to preserve geometric consistency in the underlying 3D scene structure during editing.

Existing methods attempt to mitigate this through distortion-aware designs (e.g., cube-maps or 3D spherical positional embeddings), but empirical observations suggest they still struggle to maintain geometric consistency when operating directly on ERP panoramas. Furthermore, the field is hindered by a scarcity of large-scale, high-quality training data tailored specifically for in-context panoramic tasks. A critical open question remains: How can depth priors be formulated under spherical geometry to preserve geometric consistency in in-context panoramic image generation?

Methodology

The authors propose Canvas360, a two-stage framework that integrates geometry-aware pretraining with unified in-context fine-tuning.

1. Geometry-aware Pretraining (Stage 1)

To instill strong spatial and geometric priors, the model is pretrained on a large-scale, depth-augmented dataset using a Flow Transformer architecture.

Parallel Depth Generation: The model generates RGB panoramas and spherical depth maps simultaneously. Depth is treated as an explicit geometric prior where depth corresponds to radial distance from the camera center (spherical space) rather than the Cartesian Z-axis.
Token-wise Concatenation & Positional Offsets: RGB and depth latents are concatenated at the token level. To prevent the model from conflating the two modalities, a constant offset is applied to the first dimension of the 3D Rotary Positional Embeddings (RoPE) for depth tokens, separating them from RGB tokens in positional space.
Velocity Circular Padding: To enforce spherical continuity and boundary consistency, the authors introduce velocity circular padding. Unlike standard padding that merely copies boundary columns, this method synchronizes the velocity fields of the boundary tokens (0° and 360°) with their circular counterparts before transformer computation. This explicitly supervises the model to predict boundary-consistent velocities, ensuring seamless seam alignment.
Similarity Loss Regularization: To prevent the RGB and depth predictions from collapsing into degenerate, overly similar outputs, a similarity loss is added. This penalizes excessive correlation between the predicted velocity fields of the two modalities, encouraging distinct, modality-specific representations.

2. Unified In-context Fine-tuning (Stage 2)

Building on the pretrained model, a unified framework is trained for four downstream tasks: style transfer, inpainting, outpainting, and editing.

Depth Removal: During fine-tuning, depth inputs are discarded. The model relies solely on RGB-based in-context conditions (masked images or full reference images) and text prompts, implicitly inheriting the spatial structure learned during pretraining.
Unified Representation: All tasks are formulated under a shared training tuple using token-level concatenation. Context tokens and target tokens are separated by a positional offset ( $T_c$ ), allowing the model to differentiate semantic roles while maintaining spatial correspondence.

3. Data Pipeline: Canvas360Dataset

To address data scarcity, the authors curated Canvas360Dataset, a 1M-sample dataset:

Seed Set: 100K high-quality RGB–depth panoramas derived from Matterport3D, web resources, and state-of-the-art generation models.
Synthesis Pipeline: A scalable pipeline generates 900K additional paired samples for the four downstream tasks:
- Style Transfer (200K): Synthesized using FLUX.2-dev with style-specific prompts.
- Outpainting (250K): Generated by sampling random camera parameters (yaw, pitch, FoV) to create perspective-view masks and inverse projections.
- Inpainting (250K): Created with random rectangular masks, utilizing both global scene prompts and local region prompts (generated by VLMs).
- Editing (200K): Synthesized using VLMs and grounding models (Florence2, GroundingDino) to localize objects, followed by object erasure/addition using FLUX.2 and refinement with NanoBanana.

Key Results

Extensive experiments demonstrate that Canvas360 achieves superior performance in both basic text-to-panorama generation and in-context tasks.

Quantitative Performance: On the text-to-panorama validation set, Canvas360 achieves the best FAED score (a panorama-specific fidelity metric), significantly outperforming prior methods like DiT360, HunyuanWorld, and SMGD. It also leads in IS, QA aesthetic, and NIQE scores, while remaining competitive in FID and BRISQUE.
Qualitative Improvements: Visual comparisons show that Canvas360 produces sharper details, correct polar distortions, and seamless boundary alignments compared to baselines that often suffer from blur, artifacts, or geometry-inconsistent edits.
User Study: In a study with 71 participants, Canvas360 received the highest preference scores for Boundary Continuity (31.1%), Panorama Awareness (34.1%), and Overall Quality (35.9%), outperforming Matrix-3D, HunyuanWorld, and DiT360.
Ablation Studies: Removing parallel depth generation, positional offsets, or similarity loss leads to geometric distortions, unstable training (over-darkened outputs), or degenerate modalities. Velocity circular padding is shown to be critical for boundary continuity.

Significance and Contributions

The paper claims the following contributions:

Canvas360 Framework: A two-stage approach that successfully transfers geometry-aware priors from depth-augmented pretraining to unified in-context generation, improving spatial consistency and geometric fidelity without requiring explicit geometric supervision during downstream tasks.
Geometry-aware Pretraining Strategy: A novel method utilizing parallel RGB–depth generation, regularized by similarity loss and velocity circular padding, to enforce spherical continuity and distinct modality representations.
Canvas360Dataset: The release of a 1M-scale dataset covering four distinct in-context tasks, which the authors note is the most comprehensive of its kind to date, enabling large-scale model scaling in panoramic generation.
Unified Performance: The demonstration that a single model can effectively handle diverse tasks (inpainting, outpainting, editing, style transfer) with strong generalization, surpassing prior methods in both task coverage and modeling flexibility.

The authors conclude that Canvas360 establishes a strong foundation for future scaling and broader applications in panoramic generation by effectively addressing the challenges of geometric consistency and data scarcity.