💻 computer science

DriveWeaver: Point-Conditioned Video Inpainting for Controllable Vehicle Insertion in Autonomous Driving Simulation

DriveWeaver is a novel framework for autonomous driving simulation that leverages point-conditioned video inpainting and a global-to-local hierarchical strategy to generate high-quality, temporally consistent, and geometrically accurate vehicle insertions with seamless lighting integration, thereby enabling scalable scene augmentation without relying on pre-reconstructed 3D assets.

Original authors: Junzhe Jiang, Zipei Ma, Zijie Pan, Li Zhang

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junzhe Jiang, Zipei Ma, Zijie Pan, Li Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a director filming a movie about self-driving cars. To test the car's AI, you need to create tricky situations—like a car suddenly swerving into the road or driving in a heavy rainstorm. Traditionally, directors had to build these scenes using expensive, pre-made 3D models (like digital LEGO bricks). But these models often looked fake because the lighting didn't match the real background, and if you wanted a new type of car, you had to go find or build a new 3D model first.

DriveWeaver is a new "digital magic wand" that solves these problems. Here is how it works, broken down into simple concepts:

1. The "Ghost Sketch" (Point-Conditioned Inpainting)

Instead of asking the computer to build a car from scratch, DriveWeaver uses a point cloud. Think of a point cloud as a "ghost sketch" made of thousands of tiny dots that show exactly where a car is and how it's shaped, but without any color or texture.

The Analogy: Imagine you have a blank canvas with a hole in it (the empty road). You hand the artist a transparent sheet with a dotted outline of a car drawn on it. The artist doesn't need to know what a car looks like; they just need to fill in the dots with realistic paint, shadows, and reflections that match the sunny or rainy day in the background.
The Result: DriveWeaver takes these "ghost sketches" and fills in the missing parts with a realistic car that perfectly matches the lighting and style of the scene. It's like a video editor that can seamlessly paste a car into a video, making it look like it was always there.

2. The "Long Movie" Problem (Global-to-Local Strategy)

Making a short video clip is easy, but making a long video where the car drives for a minute without glitching is hard. Old methods would make the car slowly change shape or drift away from its path, like a character in a video game who starts to look weird after running for too long.

The Analogy: Imagine writing a long story. If you write it sentence by sentence without a plan, the main character might forget their name or change their personality halfway through.
The Solution: DriveWeaver uses a two-step strategy:
1. Global Anchoring: First, it writes a rough outline of the whole story (the car's path) at a slow speed to make sure the character stays the same person from start to finish.
2. Local Interpolation: Then, it fills in the fast details between those outline points.
The Result: The car stays looking exactly the same and follows the exact path for the entire duration, no matter how long the video is.

3. The "Instant Replay" (3D Gaussian Distillation)

The video DriveWeaver creates is beautiful, but it takes a long time to generate (like rendering a high-quality movie). Self-driving cars need to see things instantly, in real-time.

The Analogy: Think of DriveWeaver as a master chef who cooks a gourmet meal (the video) in a slow kitchen. Once the meal is cooked, they take a photo of it and turn that photo into a "ready-to-eat" frozen meal that can be heated up instantly.
The Solution: DriveWeaver takes the high-quality video it just made and converts it into a 3D Gaussian representation. This is a special, lightweight 3D format that computers can render super fast.
The Result: You get the high-quality look of the slow video, but it runs fast enough for a self-driving car to use in a real-time simulation.

Why is this a big deal?

No More "Uncanny Valley": Because it paints the car directly onto the video based on the environment's lighting, the car doesn't look like a plastic toy pasted on top. It looks real.
Endless Variety: You don't need a library of 3D models. You can use point cloud data from any source (even different datasets) to generate a car. It's like having a universal translator for car shapes.
Scalable: It can create thousands of these "corner cases" (rare, dangerous scenarios) automatically to train self-driving cars to be safer, without needing humans to manually build 3D models for every single test.

In short, DriveWeaver is a tool that lets engineers "draw" cars into driving videos using simple dot-sketches, ensuring those cars look real, stay consistent over time, and can be used instantly to test how well self-driving cars handle tricky situations.

Technical Summary: DriveWeaver

Problem Statement

Autonomous driving simulation requires the insertion of foreground vehicles with predefined trajectories to enhance scene diversity and generate corner cases for testing. Current approaches face two primary limitations:

Lighting and Style Inconsistency: Methods relying on pre-reconstructed 3D assets (e.g., CAD models or mesh libraries) often struggle to harmonize the inserted foreground with the background's complex illumination and style, leading to a persistent "domain gap."
Scalability and Generalization: Existing pipelines depend heavily on labor-intensive processes, such as manual pose calibration, heuristic asset selection, and the acquisition of high-quality 3D meshes. This reliance hinders large-scale deployment and limits generalization across diverse real-world scenarios.
Temporal Instability: While some generative approaches exist, they often suffer from error accumulation and semantic drift during long-term generation, violating the strict 3D consistency required for driving simulation. Furthermore, purely generative video models are computationally prohibitive for real-time closed-loop simulation.

Methodology

DriveWeaver is a novel framework designed for controllable vehicle insertion using a point-conditioned video inpainting diffusion model. The pipeline consists of four core components:

1. Point-Conditioned Video Inpainting

The core of DriveWeaver is built upon the Wan2.1 VACE-14B video diffusion backbone, which utilizes a flow matching framework.

Input Conditions: Instead of relying on 3D meshes or reference images, the model uses rendered point cloud maps as pixel-level geometric conditions. LiDAR points are projected onto camera images, colorized, and rasterized with a fixed radius to create dense geometric guidance maps ( $I_{pc}$ ) and masks ( $M_{pc}$ ).
PointAdapter: To integrate these conditions without retraining the massive backbone, a lightweight PointAdapter is introduced. This adapter consists of a separate DiT branch that processes the concatenated point cloud maps, background masks, and inactive region RGB images. The adapter's output is projected and injected as a residual modulation into specific layers of the frozen Wan-DiT backbone. This allows the model to leverage the generative priors of the large-scale video model while strictly adhering to the geometric constraints provided by the point clouds.
Inference: The model employs Classifier-Free Guidance (CFG) to enhance realism, zeroing out point cloud conditions during unconditional sampling steps.

2. Global-to-Local Hierarchical Inpainting

To address the challenge of long-term generation (spanning hundreds of frames) without temporal drift:

Global Anchoring Stage: The model first processes a temporally subsampled, low-frame-rate sequence in a single forward pass. This establishes a consistent global identity, geometry, and lighting anchor for the entire trajectory.
Local Interpolation Stage: The original high-frame-rate timeline is partitioned into segments. The sparse anchor frames generated in the first stage serve as dense guidance to replace the sparse point cloud conditions. The model then performs conditional generation to synthesize the high-frame-rate local segments, ensuring strict 3D and temporal consistency.
Training Strategy: The model is trained in two complementary modes: GT-Free (generating from sparse conditions only) for the global stage, and GT-Aware (using ground truth frames as supervision) for the local interpolation stage.

3. 3D Gaussian Distillation

To enable real-time rendering for simulation, the generated 2D video is distilled into an explicit 3D representation:

The high-fidelity, temporally consistent video output is processed by OmniRe, a 3D Gaussian Splatting (3DGS) urban reconstruction framework.
The pipeline explicitly decouples the inserted vehicle instance from the background scene. This allows the extracted 3D Gaussian assets to encapsulate the complex illumination, shadows, and stylistic properties generated by the diffusion model.
These distilled 3D assets can then be integrated into standard 3DGS rasterization pipelines, bypassing the computationally expensive diffusion model during the simulation phase.

4. Implementation Details

Datasets: Training utilizes the Waymo Open Dataset and PandaSet, curated into approximately 10,000 samples.
Architecture: Based on Wan2.1 VACE-14B with a 3B parameter PointAdapter inserted every 5th block.
Training: The backbone is frozen; only the PointAdapter is updated. Training runs for 20,000 iterations on 8 NVIDIA H200 GPUs.
Distillation Cost: While full reconstruction takes time, optimizing only the inserted foreground vehicle against a pre-trained background 3DGS model converges in approximately 10 minutes per instance.

Key Contributions

Novel Framework: Proposes DriveWeaver, a point-conditioned video inpainting diffusion model that achieves scalable and generalizable foreground vehicle insertion using cost-effective point cloud data rather than expensive 3D assets.
Hierarchical Strategy: Introduces a global-to-local hierarchical inpainting strategy that ensures long-term temporal consistency and prevents semantic drift in extended sequences, a critical requirement for autonomous driving simulation.
Real-Time Rendering Pipeline: Demonstrates the ability to distill generated 2D videos into explicit 3D Gaussian representations, facilitating real-time rendering for downstream model training and evaluation.
Performance: Extensive experiments on Waymo and PandaSet datasets show that DriveWeaver outperforms existing state-of-the-art methods in visual realism (FID, FVD, LPIPS) and geometric consistency (PSNR, SSIM).

Results

Visual Realism: DriveWeaver significantly outperforms baselines (HUGSIM, StreetCrafter, Wan2.1 VACE-14B) in visual realism metrics. On Waymo, it achieves an FID of 15.50 (Short-term) and 17.50 (Long-term), compared to 49.06 and 56.99 for HUGSIM, respectively.
Geometric Consistency: The method maintains high geometric consistency, enabling successful 3D reconstruction. On Waymo, DriveWeaver achieves a PSNR of 34.69 (Short-term) and 32.23 (Long-term), surpassing StreetCrafter which suffers from catastrophic reconstruction failures due to temporal inconsistency.
Distillation: The distilled 3D Gaussian representation (DriveWeaver-D) incurs only marginal degradation in visual realism compared to the raw video generation while enabling real-time rendering capabilities.
Ablation Studies: Experiments confirm that removing the PointAdapter leads to catastrophic forgetting (FVD doubles), naive point projection results in blurry outputs, and reverting to autoregressive generation causes significant temporal drift.

Significance

The paper claims that DriveWeaver provides a robust tool for scalable autonomous driving scene augmentation. By replacing labor-intensive 3D asset pipelines with accessible point cloud conditions and leveraging generative priors for seamless environmental harmony, the method addresses the critical bottleneck of creating diverse, high-fidelity corner cases. The ability to generate long, consistent videos and distill them into real-time renderable 3D assets bridges the gap between high-fidelity offline generation and the requirements of online, closed-loop autonomous driving simulation.