💻 computer science

Beyond the Frame: Generating 360 Panoramic Videos from Perspective Videos

This paper introduces a novel framework for generating realistic and spatio-temporally consistent 360-degree panoramic videos from standard perspective inputs by leveraging curated 360-degree data and employing geometry- and motion-aware operations to overcome the challenges of expanding the field of view.

Original authors: Rundong Luo, Matthew Wallingford, Ali Farhadi, Noah Snavely, Wei-Chiu Ma

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rundong Luo, Matthew Wallingford, Ali Farhadi, Noah Snavely, Wei-Chiu Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie on a tiny smartphone screen. You can see the main character walking down a street, but you can't see what's happening behind them, to their left, or above their head. You are stuck in a "tunnel."

Now, imagine if you could magically pull the walls of that movie screen outward, revealing the entire world around the character—360 degrees, all the way around, like a giant bubble. That is exactly what this paper, "Beyond the Frame," is about.

The researchers built a new AI model called Argus (named after a Greek giant with a hundred eyes) that can take a standard, narrow video and expand it into a full, immersive 360-degree panoramic video.

Here is how they did it, explained with some everyday analogies:

1. The Problem: The "Tunnel Vision" AI

Current AI video generators are great at making short clips, but they suffer from "tunnel vision." If you ask them to show what's happening behind the camera, they just guess or make things up that don't match the rest of the scene. It's like trying to finish a jigsaw puzzle when you only have the center piece and no idea what the edges look like.

2. The Solution: Learning from "360-Degree" Cameras

To teach Argus how to see the whole world, the researchers didn't just use regular movies. They went fishing in a different ocean: 360-degree videos (the kind you see on YouTube where you can drag your mouse to look around).

The Analogy: Imagine you want to learn how to bake a perfect cake. Instead of just reading a recipe, you watch thousands of videos of people baking cakes from every angle. Argus learned by studying millions of these "all-around" videos to understand how the world looks when you turn your head.

3. The Magic Tricks (How It Works)

Turning a narrow video into a wide one is hard because the AI has to guess what's happening in the dark. The team used three clever tricks to make the AI smarter:

Trick 1: The "Steady Hand" (Camera Motion Simulation)
Real people holding cameras shake, tilt, and turn. If you just feed random shaky videos to the AI, it gets confused. The researchers created a "robot hand" that simulates natural human movement. This taught the AI: "Okay, when the camera tilts up, the sky moves up, and the ground moves down. Got it."
Trick 2: The "Fixed Compass" (View-Based Frame Alignment)
Imagine trying to draw a map of a city, but every time you draw a new street, you rotate the paper so North is in a different spot. It would be a mess.
The researchers made sure that no matter how the camera moved in the input video, the AI always kept the "sky" at the top and the "ground" at the bottom in its internal map. This kept the world consistent and stopped the AI from getting dizzy.
Trick 3: The "Seamless Stitch" (Blended Decoding)
When you wrap a flat map around a globe, the left edge and the right edge meet. In video, this is where the "seam" is. If the AI isn't careful, the left side of the image might look like a tree, and the right side might look like a car, creating a weird glitch where they meet.
The researchers taught the AI to generate the video twice: once normally, and once flipped 180 degrees. Then, they "blended" the two together, like mixing paint, to smooth out any rough edges. Now, the transition from left to right is invisible.

4. What Can Argus Do? (The Superpowers)

Once Argus learned to see the whole world, it unlocked some cool new abilities:

Video Stabilization without Cropping: Usually, to stop a shaky video from looking wobbly, you have to crop (cut off) the edges, making the picture smaller. Argus can stabilize the video while keeping the entire view because it knows what's happening in the "cut-off" areas.
The "Time-Travel" Camera: You can watch a video of a car driving, and then tell Argus, "Show me what the car looked like from the side," even though the original camera was only in front. Argus generates that new view for you.
Solving Mysteries (Interactive Q&A): Imagine a video of a car approaching a crosswalk. A standard AI might say, "I can't see if the car hit the crosswalk." But Argus can rotate the view to look from the side, revealing, "Ah, yes, the front bumper is definitely on the crosswalk!" It helps humans understand the scene better by letting them look around.

The Bottom Line

This paper introduces Argus, a model that acts like a magical camera lens. It takes a boring, narrow video and expands it into a full, 360-degree world that feels real and consistent. It's a huge step forward for making videos that feel less like watching a screen and more like stepping into the scene.

1. Problem Statement

The paper addresses the task of Video-to-360° Generation: taking a standard perspective video (with a limited field of view, or FoV) as input and generating a complete, temporally consistent 360° panoramic video.

Key Challenges:

Under-constrained Input: The input video only shows a narrow slice of the scene. The model must infer the spatial layout and object dynamics of the entire surrounding environment, including areas never seen in the input.
Spatio-Temporal Consistency: Unlike standard video generation, the output must maintain consistency not just over time, but also across the 360° spatial domain. Standard models often suffer from "tunnel vision" where content changes inconsistently when the view rotates back to previously observed areas.
Geometric Distortion: 360° videos are typically represented as equirectangular images, which introduce severe nonlinear distortions (especially near the poles), complicating object representation and boundary coherence.
Lack of Training Data: Existing video generation models are trained on narrow FoV data, making them poor at extrapolating full scene dynamics.

2. Methodology: Argus

The authors propose Argus, a diffusion-based model designed specifically for this task. The framework leverages a large, curated dataset of 360° videos to learn scene priors.

A. Data Curation and Training Strategy

Dataset: The authors filtered the 360-1M dataset down to ~284,000 high-quality 360° video clips.
Camera Motion Simulation: To train the model on realistic perspective inputs, they simulate perspective views from the 360° ground truth. They introduce a sampling strategy that mimics natural human camera motion (linear drift, oscillation, and noise) rather than simple static cropping. This ensures the model learns to handle complex, real-world camera trajectories.
Architecture: Based on Stable Video Diffusion (SVD), the model uses a Latent Diffusion framework. It encodes both the input (converted to equirectangular format with black padding) and the target 360° video into latent space. A U-Net denoiser is conditioned on the input video features (via CLIP) and timestamps.

B. Key Technical Innovations

To overcome the specific challenges of 360° generation, three core techniques are introduced:

View-Based Frame Alignment:
- Problem: Naively mapping input frames to the center of the equirectangular map forces the model to learn complex, varying distortions (e.g., the sky appearing at different heights).
- Solution: The authors use a SLAM framework to estimate camera poses and Euler angles relative to the first frame. They project the input frames onto a shared coordinate system. This ensures that specific scene regions (e.g., the sky, the road) remain in consistent locations across the equirectangular map, significantly simplifying the learning process.
Blended Decoding:
- Problem: Equirectangular images have a discontinuity where the left and right edges meet. Standard VAEs trained on perspective images struggle to generate coherent content across this boundary, leading to artifacts.
- Solution: The model decodes the latent representation twice: once in the original orientation and once rotated by 180°. The two resulting images are blended in pixel space using a distance-based weighting function. This ensures that artifacts from the boundary are smoothed out, as the "bad" edge of one rotation corresponds to the "good" center of the other.
Context-Aware Long Video Generation:
- To generate videos longer than the training sequence length ( $T$ ), the model is trained to predict subsequent frames conditioned on a visible context window ( $S$ frames). During inference, an iterative sampling process uses recent predictions as context for the next iteration.

3. Key Contributions

First Video-to-360° Model: Argus is the first model capable of generating realistic, coherent 360° panoramic videos from single-view perspective inputs.
Novel Training Pipeline: The introduction of camera motion simulation and view-based frame alignment allows the model to learn robust scene priors from 360° data without requiring explicit camera pose labels during inference.
Boundary Coherence: The blended decoding technique effectively solves the equirectangular boundary artifact problem, a common failure point in panoramic generation.
Dynamic Scene Understanding: The model demonstrates an ability to extrapolate object trajectories (e.g., predicting a car's path even after it leaves the frame) and maintain geometric consistency (e.g., straight lines remain straight).

4. Experimental Results

The authors evaluated Argus against baselines including PanoDiffusion (adapted for video) and state-of-the-art Video Outpainting methods.

Quantitative Performance: Argus significantly outperforms baselines in:
- Spatial Coherence: Measured by Line Consistency (straight lines in extrapolated views).
- Temporal Consistency: Measured by FVD (Fréchet Video Distance) and Motion Smoothness.
- Visual Quality: Higher PSNR and Aesthetic scores compared to outpainting methods, which degrade rapidly as the view extends from the input.
Qualitative Results:
- Argus generates seamless 360° videos where objects entering/exiting the frame do so naturally.
- It handles diverse inputs, including in-the-wild videos with complex camera motion and synthetic text-to-video inputs.
- Ablation Studies confirm that removing frame alignment leads to blurriness and inconsistency, while removing blended decoding results in visible boundary artifacts.

5. Significance and Applications

The paper demonstrates that generating full 360° videos resolves the "tunnel vision" limitation of current video AI, enabling better spatial understanding. The authors showcase several downstream applications:

Video Stabilization: Unlike traditional methods that crop the video (losing FoV), Argus stabilizes the camera while preserving the full 360° context, allowing for a larger effective field of view.
Camera Viewpoint Control: Users can interactively rotate the camera within the generated scene to view areas that were originally off-screen, aiding in scene exploration.
Dynamic Environmental Mapping: The generated 360° videos can serve as dynamic environment maps for relighting 3D objects (e.g., rendering realistic reflections on metallic spheres).
Interactive Visual Question Answering (VQA): The model enables "what-if" scenarios. For example, if a user asks, "Does the car overlap the crosswalk?" and the input view is ambiguous, the system can rotate the camera to a new angle to verify the answer, overcoming the limitations of fixed-viewpoint analysis.

Conclusion

"Beyond the Frame" presents a significant step forward in video generation by bridging the gap between limited perspective inputs and immersive 360° outputs. By leveraging 360° data priors and introducing geometry-aware architectural modifications, Argus achieves high-fidelity, temporally consistent panoramic generation, opening new avenues for VR/AR, video analysis, and interactive media.