💻 computer science

InfiniVerse: Occupancy Guided Unbounded Scene Generation for Autonomous Driving

InfiniVerse is a unified pipeline that generates realistic, controllable, and temporally coherent long-range urban scenes for autonomous driving by reconstructing 3D occupancy from a single frame, extending it autoregressively, and refining it through a hierarchical sketch-and-refine paradigm that aligns video diffusion outputs with 3D spatial representations.

Original authors: Xiaoyu Ye, Leheng Li, Xinyu Ji, Yingjie Cai, Hongda He, Xu Yan, Guanyi Zhao, Ying-Cong Chen, Bingbing Liu, Shuguang Cui, Zhen Li

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaoyu Ye, Leheng Li, Xinyu Ji, Yingjie Cai, Hongda He, Xu Yan, Guanyi Zhao, Ying-Cong Chen, Bingbing Liu, Shuguang Cui, Zhen Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an architect trying to build a never-ending, realistic movie of a city street for a self-driving car to learn from. Usually, this is incredibly hard. If you just ask a computer to "keep making the video," it eventually starts hallucinating—roads might twist into impossible shapes, buildings might vanish, or the car might drive into thin air. If you try to force it to follow a strict map, the video looks stiff and fake.

InfiniVerse is a new system that solves this by acting like a master builder who works with both a rough sketch and a high-definition camera.

Here is how it works, step-by-step:

1. The "3D Skeleton" (The Occupancy Grid)

First, the system looks at a single snapshot of a street (taken from multiple angles, like a car's cameras). Instead of just seeing a flat picture, it builds a 3D "skeleton" of the world. Think of this as a giant, invisible grid of blocks where the computer decides: "This block is a road, this one is a building, this one is empty air."

Why this matters: This gives the system a solid, physical structure to hold onto, so it doesn't lose its way as the video gets longer.

2. The "Rough Sketch" (Autoregressive Extension)

Now, imagine you want the car to drive down a new road that doesn't exist in the original photo. You draw a simple line (a "sketch") on a map showing where you want to go.

InfiniVerse uses this sketch to extend the 3D skeleton forward. It fills in new blocks of the city along your path.
At this stage, the city is still a bit blocky and low-resolution (like a Minecraft world), but it has the correct shape and layout.

3. The "High-Def Camera" (Video Generation)

Next, the system takes that blocky 3D skeleton and asks a powerful AI video generator: "Make this look like a real, photorealistic movie."

It turns the rough blocks into smooth asphalt, shiny cars, and realistic trees.
Crucially, it does this while keeping the camera moving exactly along the path you sketched.

4. The "Feedback Loop" (The Secret Sauce)

This is the most important part. Usually, AI video generators get "drunk" on their own creativity and drift away from reality after a while. InfiniVerse prevents this with a two-way conversation:

3D guides 2D: The rough 3D skeleton tells the video generator, "Stay on the road; don't make the building float."
2D guides 3D: After the video is made, the system looks at the beautiful, realistic images and projects them back into the 3D skeleton. It says, "Hey, the video shows a tree here, so update the 3D skeleton to include a tree."

This creates a closed loop. The 3D structure keeps the video from getting weird, and the realistic video keeps the 3D structure accurate. They constantly check each other's work.

The Result

The paper claims this system can generate long, continuous driving videos that stay realistic and physically consistent for much longer than previous methods.

No Magic Maps Needed: It doesn't need a pre-made, perfect map of the world. It builds the world as it goes, based on just one starting photo.
Stability: It doesn't suffer from the "drift" where the road curves into a wall or the sky turns green.
Control: You can tell it to turn left, go straight, or turn right, and it will build a realistic city that follows that path.

In short, InfiniVerse is like a self-correcting movie director that builds the set (3D) and films the scene (2D) simultaneously, constantly checking that the actors are still standing on the set and the set still looks like the movie. This allows it to create endless, realistic driving scenarios for self-driving cars to practice on.

Technical Summary: InfiniVerse

Problem Statement

The generation of realistic, controllable, and temporally coherent urban environments remains a critical, unresolved challenge in autonomous driving. While existing methods can synthesize short-range clips, they struggle with long-horizon generation.

Implicit-Autoregressive methods (e.g., video diffusion models) produce high-fidelity visuals but lack explicit 3D structure, leading to accumulated geometric drift and "hallucinations" over time.
Explicit-Prior-Guided approaches (e.g., those relying on BEV/HD maps) enforce spatial consistency but are limited by the range of input priors and struggle to render long-duration videos or maintain alignment between generated geometry and visual appearance.

There is a need for a unified pipeline that can generate long-range, 2D–3D-aligned, and controllable dynamic scenes from a single frame without relying on rigid, pre-defined maps.

Methodology: InfiniVerse

InfiniVerse is a unified framework that bridges 3D occupancy modeling and 2D video generation through a hierarchical sketch-and-refine paradigm. The system operates in two primary stages, connected by a bidirectional closed-loop feedback mechanism.

1. Image-Guided Voxel Generation (Initial Reconstruction)

Backbone: The system utilizes XCube, a large-scale 3D generative model based on hierarchical sparse voxel grids and latent diffusion.
Conditioning: Input multi-view frames are processed via an LSS (Lift-Splat-Shoot) unprojection pipeline. Features extracted from DINO-V2 are encoded into a dense 3D voxel grid.
Output: This stage reconstructs a detailed initial occupancy scene ( $O_0$ ) with fine-grained semantics and geometry, encoded into a triplane latent representation for efficiency.

2. Sketch-Guided Voxel Generation (Coarse Extension)

Mechanism: To extend the scene along arbitrary user-defined trajectories, the system employs a sketch-guided diffusion head.
Process: Given an initial grid and a trajectory sequence, the model predicts the next occupancy volume. This is decomposed into:
- Coarse-level Sketching: Generates a global scene layout at reduced resolution (e.g., $128^3$ volume) to capture the overall structure efficiently.
- Sliding Window: Uses a denoising loop with context from previous patches to ensure smooth transitions and local coherence across the trajectory.

3. Bidirectional Closed-Loop Generation (Refinement)

The core innovation is a reciprocal loop between 3D occupancy and 2D video:

Voxel-to-Video (Coarse to Fine):
- The coarse occupancy grid is converted into Multi-Plane Images (MPIs) to encode both semantic and geometric information (including occluded content).
- A video diffusion model (based on CogVideoX1.5-5B, fine-tuned with LoRA) translates these MPIs into high-fidelity, multi-view video sequences.
- Control: The system injects MPI features into the UNet decoder via 1×1 convolutions and utilizes cross-view/cross-frame attention to maintain consistency across cameras and time.
Video-to-Voxel (Refinement):
- The generated high-fidelity RGB videos are re-projected back into the 3D space.
- This acts as an image-conditioned feedback signal to refine the occupancy representation, correcting drift and enhancing geometric details.
- This creates a closed-loop refinement process where the 3D scene and 2D video mutually constrain and enhance each other.

4. Training and Control

Loss Functions: The model is trained with a unified denoising objective. A region-weighted reconstruction loss is introduced to prioritize road surfaces and dynamic actors (vehicles, pedestrians).
Text Control: High-level appearance (weather, style) is controlled via text encoders (e.g., "snowy," "sandstorm") injected into the diffusion process.
Ultra-long Rollout: A sliding-window sampler with periodic re-anchoring of features prevents drift over arbitrarily long sequences (up to 300+ frames).

Key Contributions

Unified Pipeline: Proposes InfiniVerse, a system that generates geometrically accurate and semantically coherent long-range 3D scenes from a single multi-view frame, eliminating the need for explicit HD maps or BEV priors.
Hierarchical Sketch-and-Refine Paradigm: Introduces a two-stage process that decomposes long-range generation into coarse structural sketching and fine-grained image-conditioned refinement, enhancing consistency and controllability.
Reciprocal 2D-3D Alignment: Establishes a bidirectional feedback loop where 3D occupancy guides video generation, and generated video refines the 3D representation, significantly reducing geometric drift and hallucinations.
State-of-the-Art Performance: Demonstrates superior performance in both generation quality and duration on standard benchmarks.

Experimental Results

The method was evaluated on the Waymo Open Dataset and nuScenes.

Quantitative Metrics:
- FID (Fréchet Inception Distance): Achieved 6.40 on nuScenes, significantly outperforming existing methods (e.g., Uniscene at 6.45, MagicDrive-V2 at 20.91).
- FVD (Fréchet Video Distance): Achieved 67.97, demonstrating superior temporal consistency.
- Long-Horizon Stability: As rollout length increases (up to 300 frames), InfiniVerse shows a much slower degradation trend in FID and FVD compared to baselines like Vista.
Qualitative Results:
- The system successfully generates long-duration videos with realistic appearance, weather conditions, and dynamic actors.
- It maintains strict alignment across multiple modalities: RGB, semantics, depth, occupancy, and LiDAR.
- User-defined sketches (e.g., left turn, straight lane) effectively guide the trajectory and scene layout.

Significance and Claims

The paper claims that InfiniVerse addresses the fundamental limitations of current driving scene generation by unifying explicit 3D structural reasoning with the visual realism of video diffusion.

Overcoming Drift: By coupling occupancy-guided world modeling with video generation, the framework solves the "hallucination" problem common in purely implicit autoregressive methods.
Scalability: The approach enables the synthesis of arbitrarily long driving scenarios without the constraints of pre-defined map ranges.
Application: The system provides a scalable paradigm for generating controllable urban environments, offering a promising direction for data generation, simulation, and large-scale evaluation in autonomous driving systems.

The authors emphasize that their method achieves these results starting from a single frame, without requiring ground-truth maps or explicit priors, marking a shift toward more flexible and robust scene synthesis for autonomous driving.