💻 computer science

360Anything: Geometry-Free Lifting of Images and Videos to 360°

360Anything is a geometry-free framework that leverages pre-trained diffusion transformers to lift perspective images and videos to 360° panoramas in a purely data-driven manner, eliminating the need for camera metadata while achieving state-of-the-art performance and resolving boundary artifacts through a novel Circular Latent Encoding technique.

Original authors: Ziyi Wu, Daniel Watson, Andrea Tagliasacchi, David J. Fleet, Marcus A. Brubaker, Saurabh Saxena

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ziyi Wu, Daniel Watson, Andrea Tagliasacchi, David J. Fleet, Marcus A. Brubaker, Saurabh Saxena

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a photograph taken with a standard camera. It captures a narrow slice of the world, like looking through a keyhole. Now, imagine you want to turn that single, flat picture into a full 360-degree panoramic view where you can look left, right, up, and down, as if you were standing right there in the scene.

This is what the paper 360Anything does. It takes a normal photo or video and "lifts" it into a complete, immersive 360-degree world. Here is how it works, explained simply:

1. The Old Way: Trying to Force a Square Peg into a Round Hole

Previously, computers tried to do this by using a strict mathematical map. They would take the narrow photo and try to mathematically stretch and warp it onto a giant, flat map of the world (called an "equirectangular projection").

To do this, the computer needed to know the exact details of the camera that took the photo: How wide was the lens? Was the camera tilted? Which way was it pointing?

The Problem: Most photos on the internet (or from your phone) don't have this information. If the computer guesses wrong, the result looks broken or distorted. It's like trying to assemble a puzzle without the picture on the box, relying only on a manual you don't have.

2. The New Way: Learning by Example (The "360Anything" Approach)

The authors of this paper say, "Let's stop trying to do the math manually." Instead, they built a system called 360Anything that learns the relationship between a narrow photo and a full panorama simply by looking at thousands of examples.

The Analogy: Imagine teaching a child to draw a full room based on a small sketch. Instead of giving them a ruler and protractor (the camera math), you show them thousands of pairs of "small sketch" and "full room" drawings. Eventually, the child learns the pattern intuitively. They don't need to know the exact angle of the camera; they just know, "If I see a door here in the small sketch, the rest of the wall must continue like this."
The Result: 360Anything works on "in-the-wild" data—photos and videos taken by anyone, anywhere, with no special settings. It figures out the camera angle and position on its own.

3. Fixing the "Seam" Problem

When you wrap a flat image around a sphere to make a 360-degree view, the left edge and the right edge have to meet perfectly. If they don't, you see a ugly "seam" or a crack in the image, like a poorly stitched blanket.

The Old Fix: Previous methods tried to fix this after the image was made, using tricks to blur the seam or rotate the image during generation.
The 360Anything Fix: The authors realized the problem started before the image was even made. The computer's internal "translator" (called a VAE) was treating the edges of the image like a wall, putting a "zero" (a blank space) next to them. This created a glitch.
The Solution: They changed the translator to use "Circular Padding." Imagine the image is a loop of a belt. Instead of putting a blank wall at the end, the computer connects the right edge directly to the left edge before processing. This ensures the "seam" is smooth from the very beginning, resulting in a perfect, invisible circle.

4. What Can It Do?

The paper demonstrates that this method is incredibly powerful:

For Photos: It can take a single photo and generate a full 360-degree image that looks realistic, even if the original photo was taken at a weird angle or with a wide-angle lens.
For Videos: It can take a short video clip and turn it into a full 360-degree video. If you walk forward in the video, the system generates the world ahead of you. If you look up, it generates the sky.
3D Reconstruction: Because the generated 360-degree videos are so consistent, you can use them to build a 3D model of the room or street. You can then "fly through" the scene in a computer, exploring it from angles that weren't in the original video.

5. Why It's a Big Deal

The paper claims that 360Anything is better than previous methods, even those that did have the perfect camera information. By removing the need for camera data and fixing the seam issue at the root, it creates cleaner, more accurate, and more immersive 3D worlds from simple, everyday photos and videos.

In short: It turns a flat, limited view of the world into a complete, explorable 360-degree experience, without needing a manual or a map to get there.

Technical Summary: 360Anything

Problem Statement

The generation of photorealistic 3D worlds from perspective images and videos is a critical frontier for immersive applications in robotics, AR/VR, and gaming. While recent advances in visual generative models have improved perspective view synthesis, standard generators produce narrow fields of view (FoV), limiting their utility for creating fully immersive 360° environments.

Existing approaches for lifting perspective inputs to 360° panoramas typically rely on explicit geometric alignment. These methods project the perspective input into the Equirectangular Projection (ERP) space to provide a pixel-aligned conditioning signal. However, this strategy necessitates known camera metadata (Field-of-View, yaw, pitch, roll). In "in-the-wild" scenarios, such calibration data is often absent or noisy. Consequently, current models struggle with uncalibrated inputs or fail when relying on brittle external estimators. Furthermore, generated panoramas often suffer from seam artifacts at the ERP boundaries, a problem prior works attempt to mitigate with inference-time tricks rather than addressing the root cause.

Methodology

The authors propose 360Anything, a geometry-free framework built upon pre-trained Diffusion Transformers (DiT) that eliminates the need for explicit camera information.

1. Geometry-Free Sequence Concatenation

Unlike prior works that enforce spatial correspondence by projecting inputs into ERP space, 360Anything treats the perspective input and the target panorama simply as token sequences.

Mechanism: The perspective input ( $X_{pers}$ ) is encoded into latent tokens ( $x_{pers}$ ) and concatenated with the noisy target latent tokens ( $y^t_{equi}$ ) along the sequence dimension.
Learning: The DiT performs global self-attention on this combined sequence. The model learns the perspective-to-equirectangular mapping purely from data, implicitly inferring camera intrinsics and extrinsics to "place" the input onto a 360° canvas.
Canonicalization: To ensure consistent training and generation, the framework enforces a Canonical Coordinate constraint. The model is trained to generate panoramas in a standard, gravity-aligned upright orientation, regardless of the input camera's pose. This requires a two-stage data pre-processing pipeline for real-world videos:
1. Stabilization: Using COLMAP to estimate per-frame poses and rotate frames to eliminate inter-frame rotation.
2. Gravity Alignment: Using GeoCalib to estimate the global gravity direction and rotate the video so gravity aligns with the vertical axis.

2. Circular Latent Encoding for Seam-Free Generation

The paper identifies the root cause of seam artifacts in ERP generation as zero-padding in the convolutional layers of the Variational Autoencoder (VAE) used for latent encoding. Standard zero-padding introduces boundary discontinuities in the latent representation, which the diffusion model learns to replicate.

Solution: The authors propose Circular Latent Encoding. Before encoding, columns are cropped from the left and right of the panorama image and padded to the opposite sides (circular padding). After encoding, the latents corresponding to these padded regions are dropped.
Result: This ensures the latent representation possesses circular continuity, eliminating the discontinuity that causes seams without introducing overhead to the sequence length or inference time.

Key Contributions

360Anything Architecture: A novel DiT-based framework for "in-the-wild" perspective-to-panorama generation that implicitly infers camera parameters, removing the dependency on camera calibration.
Root Cause Analysis of Seams: Identification of VAE zero-padding as the source of seam artifacts and the introduction of Circular Latent Encoding as a principled, training-stage remedy.
State-of-the-Art Performance: The method achieves superior results on both image and video generation benchmarks, outperforming baselines that utilize ground-truth camera information.
Zero-Shot Geometric Understanding: The model demonstrates competitive accuracy in zero-shot camera FoV and orientation estimation, proving it learns deep geometric correspondences despite not using explicit metadata.

Experimental Results

Panoramic Image Generation

Evaluated on Laval Indoor and SUN360 datasets, 360Anything outperforms baselines (including CubeDiff, PanoDiffusion, and Diffusion360) across visual quality metrics (FID, KID, FAED).

FAED Improvement: The model reduces the FAED error by nearly 50% compared to the state-of-the-art, indicating significantly better global geometry and 360° consistency.
Qualitative: Unlike cubemap-based methods that stretch objects when input FoV differs from 90°, 360Anything correctly infers FoV and orientation, producing undistorted objects.

Panoramic Video Generation

Evaluated on the Argus dataset using both simulated and real-world camera trajectories.

Metrics: 360Anything achieves the best scores in PSNR, LPIPS, FVD, and VBench metrics (Imaging Quality, Aesthetic Quality, Motion Smoothness).
Consistency: The model generates stably canonicalized videos that remain upright and distortion-free even with large camera motion, whereas baselines often produce rotated or distorted outputs.
3D Reconstruction: The generated panoramic videos enable high-quality 3D scene reconstruction via 3D Gaussian Splatting (3DGS), demonstrating strong geometric consistency suitable for fly-through exploration.

Zero-Shot Camera Calibration

Despite being trained solely for outpainting, the model achieves competitive results in zero-shot camera parameter estimation:

FoV Estimation: Ranks among the top-3 on NYUv2, ETH3D, and iBims-1, outperforming several supervised baselines and lagging only slightly behind specialized SOTA methods (DUSt3R, MoGe).
Orientation Estimation: Outperforms most supervised baselines on MegaDepth and LaMAR for roll and pitch estimation, with errors only ~0.5 degrees higher than the SOTA GeoCalib.

Significance and Claims

The paper claims that explicit geometric alignment is unnecessary for high-quality panorama generation. By leveraging the representational power of Diffusion Transformers and treating the task as a sequence-to-sequence problem, 360Anything achieves robustness in "in-the-wild" scenarios where calibration is impossible.

The work demonstrates that:

Data-driven learning of geometric relationships can surpass methods relying on explicit, often noisy, geometric priors.
Seam artifacts are an architectural issue in the latent space, not just an inference artifact, and can be solved via Circular Latent Encoding.
The framework possesses deep geometric understanding, capable of inferring camera parameters and maintaining 3D consistency (enabling 3DGS reconstruction) without explicit supervision.

The authors conclude that this approach enables the scaling of model and data for immersive 3D world generation, moving beyond the limitations of perspective-only generators and calibration-dependent pipelines.