HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles

HorizonForge is a unified framework that enables photorealistic, controllable driving scene generation with arbitrary trajectories and vehicles by combining editable Gaussian-Mesh representations and noise-aware video diffusion, significantly outperforming existing methods in fidelity and consistency while introducing the HorizonSuite benchmark for standardized evaluation.

The Gist

Imagine you are watching a movie of a car driving down a busy street. Now, imagine you want to change the story: maybe the main car needs to take a sharp left turn instead of going straight, or you want to magically insert a bright red sports car into the traffic, or perhaps you want to make a slow-moving truck disappear entirely.

Doing this in real life is impossible. Doing it in a computer simulation is usually very hard because the computer either makes the scene look fake (like a cartoon) or it breaks the physics (the new car floats through the air).

HorizonForge is a new tool that solves this problem. Think of it as a "Magic Editing Kit" for driving videos.

Here is how it works, using some simple analogies:

1. The "Digital Clay" (3D Assets)

Most video editors just work with flat pictures (2D). If you try to move a car in a flat picture, it looks weird and stretched.

HorizonForge is different. First, it takes the original video and turns it into 3D "Digital Clay."

• The Analogy: Imagine taking a photo of a room and turning it into a virtual room made of millions of tiny, glowing dust particles (called Gaussian Splats). These particles hold all the colors and shapes of the real world.
• Why it matters: Because the scene is now 3D "clay," you can reach in, grab a car, move it to a new spot, or spin it around, and the computer knows exactly how the light should hit it from every angle. It's like sculpting with clay instead of painting on a canvas.

2. The "Ghost Writer" (Language & Trajectories)

How do you tell the computer what to do? You don't need to be a coder.

• The Analogy: You can just talk to the scene or draw a line on a map.
  • Voice: "Put a gray sedan right in front of that SUV."
  • Map: "Make the main car turn left here."
• The system understands these instructions. If you ask for a specific car, it can even generate a 3D model of that car from scratch (like a 3D printer in the computer) and drop it into the scene perfectly.

3. The "Reality Polisher" (Video Diffusion)

When you move the 3D clay around, the raw result might look a little glitchy or "digital." It needs a final polish to look like a real movie.

• The Analogy: Think of a Video Diffusion Model as a super-smart film editor and special effects artist combined.
• It looks at your "glitchy" 3D edit and says, "Okay, I see you moved the car. Now, let me fix the shadows, smooth out the motion, and make sure the background doesn't flicker."
• Crucially, it remembers the flow of time. It ensures that if a car turns left, the wheels spin and the background blurs naturally, just like in a real movie. It doesn't just fix one frame; it fixes the whole sequence so it feels real.

Why is this a big deal?

Currently, testing self-driving cars is dangerous and expensive. You can't easily create a "what if" scenario where a child runs into the street or a truck brakes suddenly.

• The Old Way: You had to build a fake city in a video game (which looks fake) or wait years to catch a rare accident on camera.
• The HorizonForge Way: You take a normal video, tell the computer "Make the truck brake hard," and it instantly creates a photorealistic, physics-perfect video of that exact event.

The "Magic" Results

The paper tested this against other methods and found:

• It looks real: Humans preferred HorizonForge over other methods 91% of the time.
• It's consistent: The cars don't flicker or change shape as they move.
• It's flexible: You can change the path of the driver, add new cars, or remove obstacles, all with simple instructions.

In short: HorizonForge is like having a time machine and a director's chair for driving videos. It lets you rewrite the rules of the road safely, instantly, and with a level of realism that makes it look like you were actually there.

Technical Summary

1. Problem Statement

Autonomous driving systems require extensive testing in "long-tail" scenarios (rare, safety-critical events like aggressive lane changes or sudden braking) to ensure safety. While real-world data collection is vast, capturing these rare events is expensive and difficult. Existing simulation methods face a trade-off:

• Reconstruction-based methods: Offer high geometric fidelity but struggle to generalize to unseen regions or synthesize novel trajectories/agents.
• Generation-based methods: Can hallucinate new content but often fail to preserve scene structure, appearance, and physical consistency, leading to stochastic motion and artifacts.
• Hybrid methods: Often rely on complex architectures and costly per-trajectory optimization, limiting scalability.

The Goal: Create a unified framework that achieves photorealism and precise controllability simultaneously, allowing for the editing of driving scenes with arbitrary trajectories and the insertion of any vehicle via natural language, without requiring per-scene optimization.

---

2. Methodology: HorizonForge

HorizonForge is a unified framework that reconstructs scenes into editable 3D representations and renders them back into video using a noise-aware diffusion process. The pipeline consists of two main stages:

A. 3D Assets Harvesting

The system first reconstructs the input driving video into editable 3D assets to support flexible manipulation in 3D space.

• Gaussian Splats (3DGS): The core representation for existing vehicles and the scene. The authors extract high-quality 3D Gaussian Splat representations (using OmniRe) because they encode richer appearance cues and continuous density compared to sparse point clouds or bounding boxes. This allows for precise editing of existing geometry.
• Mesh Harvesting & Insertion: To handle the insertion of new vehicles (which 3DGS cannot easily generate from scratch), the framework uses a mesh-based pipeline:
  • Harvesting: Extracts meshes from real data (Waymo) using ground-truth bounding boxes and LiDAR, refining them with segmentation (SAM) and inpainting (Pix2Gestalt) before generating a mesh via Hunyuan3D.
  • Alignment: A GPT-based reasoning module aligns the generated mesh's pose and scale with the target 3D bounding box to ensure physical validity.
  • Text-to-Mesh: At inference, natural language prompts are converted to reference images, which generate 3D meshes via Hunyuan3D. These are then aligned and inserted into the scene.

B. Video Rendering (Diffusion)

Once the 3D assets are repositioned according to target trajectories, they are rendered into 2D. However, raw 3D rendering often contains artifacts.

• Noise-Aware Video Diffusion: The system employs a Video Diffusion Model (VDM) (based on CogVideoX and TrajectoryCrafter) to refine the rendered frames.
• Training Strategy:
  • Cycle-Reconstruction: To train the VDM to remove Gaussian Splat artifacts, the system creates "noisy" pairs by perturbing trajectories, rendering them, and then training the model to restore the original clean video.
  • Mesh Integration: For mesh insertion, the model is trained on pairs where Gaussian assets are randomly replaced with meshes, teaching the VDM to bridge the photorealism gap between synthetic meshes and real imagery.
• Temporal Consistency: Unlike image-based diffusion, the VDM leverages temporal priors to ensure coherent motion and stability across frames, preventing flickering.

---

3. Key Contributions

1. HorizonForge Framework: A simple yet powerful paradigm combining editable 3D Gaussian Splats and Meshes with a standard Video Diffusion Model. It supports fine-grained 3D manipulation (lane changes, turns) and language-guided object insertion.
2. Mesh Harvesting & Insertion Pipeline: A novel method to insert any vehicle described by text into a driving scene, ensuring geometric alignment and physical validity without per-trajectory optimization.
3. HorizonSuite Benchmark: A comprehensive evaluation suite covering diverse tasks (ego-vehicle and agent-level manipulations, insertions, removals) with fine-grained metrics (FID, FVD, VIMS, BAS, OSR).
4. Design Principles Discovery:
   • 3D Representations Matter: Gaussian Splats significantly outperform point clouds and bounding boxes in preserving appearance details and enabling accurate edits.
   • Temporal Priors Matter: Video diffusion models are essential for coherent synthesis, outperforming image-based diffusion in maintaining temporal consistency.

---

4. Results

The authors evaluated HorizonForge against state-of-the-art baselines (StreetCrafter, Difix3D, OmniRe) on the HorizonSuite benchmark.

• Quantitative Performance:
  • FID (Fréchet Inception Distance): HorizonForge achieved 33.19 (overall), a 25.19% improvement over the second-best method.
  • FVD (Fréchet Video Distance): Significantly lower scores, indicating better temporal realism.
  • VIMS (Vehicle Identity Matching): Highest scores, indicating better preservation of vehicle identity during motion.
  • OSR (Operation Success Rate): Achieved 7.00 for insertion/removal tasks, outperforming all baselines.
• User Study: In a blind comparison involving 501 samples, HorizonForge was preferred 91.02% of the time over the next best method, confirming superior photorealism and controllability.
• Downstream Utility: Augmenting real training data with HorizonForge-generated edited scenes improved 3D object detection (BEVFormer) performance by 10–30% relative gains across IoU thresholds, proving the data's informational value.

---

5. Significance

• Scalability for Safety: HorizonForge provides a scalable solution for generating rare, safety-critical driving scenarios, which is crucial for validating autonomous driving algorithms without the cost and risk of real-world data collection.
• Unified Paradigm: It bridges the gap between reconstruction (geometry) and generation (appearance), offering a simpler alternative to complex hybrid models that require heavy optimization.
• Physical Grounding: By integrating explicit 3D control (trajectories) with generative modeling, the system ensures that edits are physically plausible (e.g., vehicles follow valid paths, no spatial conflicts).
• Standardization: The introduction of HorizonSuite establishes a unified protocol for evaluating controllable driving simulation, addressing the lack of standardized benchmarks in this field.

In conclusion, HorizonForge demonstrates that combining explicit 3D representations (Gaussian Splats/Meshes) with temporal diffusion models is the key to achieving high-fidelity, controllable, and scalable driving scene synthesis.