🤖 machine learning

Extended Field of View Analysis for VideoGAN-based Trajectory Generation

This paper presents an enhanced VideoGAN framework for generating realistic and diverse vehicle trajectories in large-scale traffic scenes by improving semantic representation, adopting graph-based trajectory extraction, expanding the field of view, and introducing a quantitative evaluation for hallucinations and object permanence, all while maintaining efficiency and coherence for downstream automated driving tasks.

Original authors: Annajoyce Mariani, Kira Maag, Hanno Gottschalk

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Annajoyce Mariani, Kira Maag, Hanno Gottschalk

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to drive a car. You can't just give it a rulebook saying "stop at red lights" and "stay in the lane," because real traffic is messy, unpredictable, and full of human quirks. To get a robot to drive like a human, it needs to understand the feeling of the road, not just the rules. This is where a branch of computer science called "generative artificial intelligence" comes in. Think of these AI models as digital artists that have studied millions of videos of traffic. Instead of just memorizing a single route, they learn the "vibe" of how cars move, how they weave through traffic, and how they react to signals. The goal is to create a simulator that can dream up new, realistic traffic scenes on the fly. This is crucial because before we let robots drive our cars, we need to test them in millions of different scenarios—some safe, some dangerous, some weird—to make sure they won't crash.

The paper you are about to read tackles a specific challenge in this dream: how do we make these AI "dreams" of traffic bigger and more complex without the computer taking forever to think? The researchers built on a previous idea where an AI watches a bird's-eye-view video of traffic (like looking down from a drone) and learns to generate new videos of cars moving. They wanted to see if they could stretch the camera's view to cover a much wider area of the road and still keep the cars behaving realistically. They found that by tweaking how they colored the traffic scenes and using a smarter way to track the cars, their AI could generate long, complex traffic scenes that look and act very much like the real thing, all while running incredibly fast.

The Paper's Story: Teaching AI to Dream Bigger Traffic

The researchers, Annajoyce Mariani, Kira Maag, and Hanno Gottschalk, set out to upgrade a system that generates traffic videos. Imagine you have a magic camera that looks down at a highway. In the past, this camera could only see a small patch of road, maybe 15 meters long. The AI learned to predict what would happen in that tiny patch. But real driving happens on long stretches of road where cars enter and exit, and where interactions happen far away. The team asked: "What if we make the camera's view much wider? Can the AI still keep track of everything without getting confused or hallucinating?"

To answer this, they didn't just turn up the zoom; they upgraded the whole toolkit. First, they changed the "language" the AI speaks. Instead of using standard colors, they switched to a special color system called "Lab," which is like giving the AI a set of high-contrast highlighters. This makes it much easier for the computer to tell the difference between a car, a traffic light, and the road itself, even when they are close together.

Next, they fixed a major headache: tracking the cars. In the old system, the AI would sometimes get confused and think two cars merged into one, or that a car suddenly split into two. To fix this, the team built a "graph" for the traffic. Imagine every car is a dot, and every time a car moves to the next frame of the video, you draw a line connecting the old dot to the new one. If the lines cross or get tangled, the system knows something is wrong. This graph-based method acts like a super-organized librarian, ensuring that every car keeps its identity from start to finish, even if it disappears behind a building and comes back out later.

They tested this new system by training it on a massive dataset of real driving videos from the Waymo Open Motion Dataset. They taught the AI to generate traffic scenes with three different field-of-view sizes: 15 meters, 20 meters, and 25 meters. They wanted to see if the AI could handle the larger, more crowded scenes without making up fake cars or making real cars vanish.

The results were impressive. The AI successfully generated traffic scenes that looked statistically realistic. When they looked at the data, the cars in the generated videos had speeds, accelerations, and distances between them that matched real-world traffic almost perfectly. For example, the cars kept safe distances, and they slowed down for red lights just like real drivers do. The system was also incredibly fast. It could generate a 20-second traffic scene in less than 20 milliseconds. To put that in perspective, that's faster than the blink of an eye, meaning this technology could potentially run in real-time on a car's computer to help it plan its next move.

However, the researchers were careful not to claim perfection. They did find some small "hallucinations," or glitches, especially in the largest 25-meter scenes. Occasionally, a car might slowly fade away at a red light, or a traffic light might change color slightly weirdly as a car passes under it. But these were rare. The team measured how often cars appeared out of nowhere or disappeared, and found that in the best versions of their model, these errors happened less than 1% of the time in the smaller scenes, and only slightly more in the larger ones. Crucially, they found that most of the "appearing" and "disappearing" events were actually just cars entering or leaving the view, which is normal, rather than the AI making things up.

The paper also compared their method to other types of AI, like "diffusion models," which are famous for making beautiful, high-quality images. While those models might make prettier pictures, they are very slow, taking seconds or even minutes to generate a single scene. The researchers argued that for driving, speed and reliability are more important than artistic perfection. Their video-based GAN (Generative Adversarial Network) was a "workhorse" that could churn out long, consistent videos quickly, making it a much better fit for the split-second decisions needed in automated driving.

In the end, the study suggests that by expanding the field of view and improving how the AI tracks objects, we can create much more realistic and complex traffic simulations. These simulations are safe, diverse, and fast enough to be used for training self-driving cars to handle the chaos of the real world. The researchers conclude that this approach is a scalable and efficient way to generate the endless variety of traffic scenarios needed to make automated driving safe for everyone.

Technical Summary: Extended Field of View Analysis for VideoGAN-based Trajectory Generation

Problem Statement

Realistic and diverse trajectory generation is a critical component for enabling higher levels of vehicle automation, particularly for prediction, planning, and simulation tasks. While rule-based and classical learning methods often struggle to capture the complexity of traffic behavior, deep learning offers a promising alternative. However, existing approaches face specific challenges:

Discrete vs. Spatial Modeling: Traditional trajectory prediction often treats the problem as forecasting a discrete set of sequential paths, which can struggle with variable-length sequences (agents entering/exiting) and complex spatial interactions. Conversely, spatial approaches like occupancy grids handle layout constraints well but often lack the generative modeling advantages of capturing full data distributions.
Computational Efficiency: While diffusion models have shown high visual fidelity and diversity, their inference times (often seconds to minutes per scene) make them less suitable for real-time trajectory proposal compared to Generative Adversarial Networks (GANs).
Scalability and Hallucinations: Extending generative models to larger fields of view (FOV) to capture long-range dependencies introduces risks of visual artifacts, such as object permanence failures (disappearing/splitting agents) and inconsistent spatial relationships, which are difficult to quantify.

Methodology

The authors propose an end-to-end pipeline that generates traffic scenarios as semantic Bird's-Eye-View (BEV) videos and extracts trajectories from them, building upon previous work [28] with several key technical enhancements:

1. Semantic Representation (Rasterization)

Instead of the HSV-based encoding used in prior work, the authors adopt a Lab color space for rasterizing trajectory data into low-resolution BEV semantic videos.

Rationale: The Lab space (Lightness, Green-Red, Blue-Yellow) provides superior separation between semantic categories (road markings, traffic lights, ego vehicle, other vehicles) compared to RGB or HSV.
Encoding:
- Road Centerlines: Rendered as gray continuous lines ( $L \in [30, 90]$ , $a, b \approx 0$ ).
- Traffic Lights: Rendered as red, green, or yellow circles ( $2\times2$ m) based on specific ellipsoidal ranges in Lab space.
- Vehicles: The ego vehicle is black (low $L$ ); other agents are sampled from a blue-pink range (negative $b$ , positive $L$ ).
Input: The system processes data from the Waymo Open Motion Dataset, converting world coordinates into image coordinates centered on a reference agent.

2. Generative Model

The framework utilizes a VideoGAN (specifically the low-resolution branch of LongVideoGAN [3]) rather than a diffusion model.

Choice Justification: While diffusion models offer higher visual fidelity, VideoGANs provide significantly faster inference (single forward pass) and training stability, which is crucial for real-time applications. The chosen architecture is designed to generate dynamically coherent scenes of arbitrary length.
Training: The model is trained on 60k video sequences (128 frames each, ~12.8s) for 200k steps on 80GB A100 GPUs.

3. Trajectory Extraction

A novel graph-based association method replaces the previous cost-matrix matching procedure.

Object Detection: Frames are processed using Lab-space color filtering to create masks for traffic lights and vehicles. An inpainting step is applied to handle occlusions (e.g., vehicles passing under traffic lights).
Graph Construction: Each detected object in a frame becomes a node. Edges are established between nodes in adjacent frames if they share the same object type and exhibit spatial overlap. Edge intensity is proportional to the overlap.
Advantage: This approach naturally handles variable-length trajectories and agents entering/leaving the scene without assuming a fixed set of tracked agents.

4. Evaluation Framework

The authors introduce a quantitative framework to assess "hallucinations" and physical plausibility:

Hallucination Metrics: Analyzing the trajectory graph to detect abnormal node degrees (e.g., nodes with two parents indicating merging, or two children indicating splitting).
Statistical Metrics: Comparing distributions of speeds, accelerations, inter-vehicle distances, and Time-to-Collision (TTC) between generated and real data.
Traffic Light Interaction: Assessing vehicle behavior (stopping at red, crossing green) relative to traffic light states.

Key Contributions

Improved Semantic Encoding: Introduction of Lab color-space encoding to enhance the separation of semantic categories, facilitating more accurate object extraction from generated videos.
Graph-Based Trajectory Extraction: Development of a graph-based tracking method that improves the robustness of trajectory extraction from semantic videos, maintaining inference times below 20 ms for scenes of up to 20 seconds.
Systematic FOV Analysis: A comprehensive investigation of scene generation across three longitudinal ranges (15 m, 20 m, and 25 m), demonstrating the model's ability to scale to larger, more complex traffic contexts.
Quantitative Hallucination Assessment: Proposal of a specific metric to measure visual artifacts (appearing, disappearing, merging, splitting vehicles) by analyzing the topology of the extracted trajectory graphs.

Results

Experiments were conducted on the Waymo Open Motion Dataset with three FOV settings (15 m, 20 m, 25 m).

Visual Quality & Hallucinations:
- Generated scenes showed qualitative similarity to training data, with smooth dynamics and aligned vehicle paths.
- Object Permanence: The model successfully handled agents leaving and re-entering the frame.
- Hallucination Rates: For well-trained checkpoints, merging and splitting rates were negligible (<1% for 15 m and 20 m FOVs). "Boundary-excluded" metrics (excluding events near frame edges) showed even lower rates of artificial appearance/disappearance, suggesting the model learns realistic scene visibility changes rather than spawning agents.
Frame-Level Statistics:
- The distributions of object footprint areas, inter-vehicle distances, and the number of vehicles/traffic lights in generated scenes closely matched real data across all FOVs.
- Generated scenes maintained appropriate safety distances between agents.
Dynamics:
- Distributions of relative speeds and accelerations (including log-scale analysis) aligned accurately with real data.
- Time-to-Collision (TTC) distributions indicated that generated trajectories were statistically as safe as real ones, with no increase in low-TTC (high-risk) events.
Traffic Light Interaction:
- Vehicles in generated scenes correctly slowed/stopped at red lights and crossed green lights, with speed distributions matching real-world observations.
Efficiency:
- Inference time was approximately 20 ms for a 15-second scene, scaling linearly with sequence length (reaching 150 ms for a 2-minute scene). This performance is deemed competitive for real-time downstream tasks.

Significance and Claims

The paper claims that video-based GANs remain an efficient and scalable approach for realistic trajectory generation, even when extended to substantially larger traffic scenes (up to 25 m longitudinal FOV).

Scalability: The framework generalizes to larger and more complex scenes while maintaining statistically realistic trajectories and coherent spatial relationships.
Practicality: With inference times below 20 ms for scenes of up to 20 seconds, the method is suitable for downstream tasks in automated driving such as prediction, planning, and simulation, where computational latency is a constraint.
Validity of Visual Generation: The results support the hypothesis that learning traffic dynamics through purely visual semantic video generation (without explicit rules or conditions) is a viable strategy for capturing complex multi-agent interactions and long-range dependencies.

The authors conclude that while larger FOVs introduce slightly more statistical variance (e.g., in object permanence metrics), the overall system demonstrates robustness, making it a strong candidate for generating diverse and safe traffic scenarios for automated driving systems.