VideoGAN-based Trajectory Proposal for Automated Vehicles

Technical Summary: VideoGAN-based Trajectory Proposal for Automated Vehicles

Problem Statement
The core challenge addressed in this work is the generation of realistic, statistically accurate trajectory options for automated vehicles (AVs). While trajectory prediction and planning are pivotal for highly automated operation, existing methods face limitations. Classical rule-based approaches struggle to model the complex, high-dimensional, and multimodal distributions governing the joint behavior of multiple agents in interactive environments. Conversely, while deep learning (DL) models, particularly generative models like Diffusion models and Generative Adversarial Networks (GANs), excel at capturing diverse futures, many spatially-grounded approaches (which handle layout constraints and contextual cues well) rely on attention-based architectures that do not fully leverage the distributional modeling benefits of generative models. Furthermore, there is a need for methods that can handle variable-length trajectories (agents entering/exiting scenes) and maintain physical realism regarding safety distances and traffic signal compliance without incurring the high computational costs associated with diffusion models.

Methodology
The authors propose a three-stage pipeline that combines spatially-grounded representation with video generation:

1. Rasterization of Abstract Data:
   The pipeline converts abstract traffic data from the Waymo Open Motion Dataset into low-resolution Bird's-Eye View (BEV) occupancy grid videos. This process involves:

   • Extracting road centerlines, time-varying traffic light states, and agent data (center coordinates, heading, bounding box size).
   • Rendering these elements into image coordinates centered on the ego vehicle. Road centerlines are rendered as gray lines; traffic lights as colored circles; and agents as rectangles.
   • The ego vehicle is rendered in black, while other agents are assigned random colors from an HSV subspace to avoid confusion with traffic lights or road markings.
   • The output is a dataset of low-resolution BEV videos (e.g., 54 × 96 pixels) representing traffic scenes.

2. VideoGAN Training:
   A video generative model, specifically a VideoGAN (based on LongVideoGAN), is trained on the rasterized occupancy grid videos. The authors selected GANs over diffusion models to prioritize faster training and inference times, noting that while diffusion models offer higher visual fidelity, this is less critical for trajectory proposal where physical plausibility and driving feasibility are the primary metrics. The model learns to generate dynamically coherent scenes of arbitrary length.

3. Trajectory Extraction:
   Abstract trajectory data is extracted from the generated videos using a post-processing pipeline:

   • Object Detection: Frames are converted to HSV color space. Masks are applied to isolate traffic lights (red, green, yellow) and agents. A specific masking technique replaces traffic light colors with the underlying occluded color (e.g., vehicle or road) to improve detection of agents passing beneath lights. Morphological opening smooths contours, and spatial features (center, size, shape) are extracted.
   • Object Matching: Identities are tracked across frames using a cost matrix based on spatial proximity, color similarity, aspect ratio, and type consistency.
   • Trajectory Enrichment: Persistent IDs are assigned, and instantaneous speed and acceleration relative to the ego vehicle are computed via interpolation.

Key Contributions

• Generative and Spatially-Grounded Pipeline: The authors present a method that leverages the distributional modeling of generative models while maintaining the spatial layout constraints of BEV representations.
• Customizable Rasterization: A pipeline to convert abstract traffic data into low-resolution BEV occupancy grid videos, allowing for high customization of scene composition (e.g., increasing the frequency of turning scenarios).
• Extraction Procedure: A developed procedure to extract abstract data regarding traffic lights and road users from the generated video predictions using frame-by-frame detection and matching.
• Efficiency: The system achieves inference times of approximately 20 ms for a 15-second scene, making it suitable for real-time applications, with training completed within 100 GPU hours.

Results
The authors evaluated the generated trajectories against ground truth from the Waymo Open Motion Dataset using both qualitative and quantitative metrics:

• Qualitative Analysis: Generated scenes exhibited realistic map lines, road user dynamics, and traffic light interactions. Agents were observed waiting at red lights, proceeding on green lights, and maintaining safety distances. The model successfully modeled complex behaviors such as turning at intersections and agents entering/exiting the scene, though occasional identity morphing (splitting or merging) was observed near crossings.
• Quantitative Analysis (Static Parameters):
  • Object Sizes: The distribution of agent sizes in generated videos was bimodal and comparable to real data, though generated vehicles were on average 10% larger.
  • Distances and Density: The average center-to-center distance between agents and traffic density (number of objects per frame) were remarkably similar between real and generated scenes.
  • Unidentified Objects: The distribution of unrecognized objects was consistent, reinforcing the visual realism of the frames.
• Quantitative Analysis (Dynamics):
  • Speed and Acceleration: Statistical distributions of relative speeds and accelerations matched the training data, capturing overtaking, co-moving, and opposing traffic behaviors.
  • Safety: Minimum edge-to-edge distances confirmed that road users maintained appropriate separation.
• Traffic Signal Compliance: Analysis of ego vehicle speed relative to traffic lights showed that the model learned to stop at red lights (speed ~0 km/h) and proceed at appropriate speeds for green lights, demonstrating compliance with dynamic traffic signals.

Significance and Claims
The paper claims to demonstrate that video generation models can produce traffic scenes and trajectories that are both realistic and safe in terms of agent interactions and compliance with dynamic traffic signals. By achieving inference speeds of 20 ms, the authors position their method as a viable candidate for real-time trajectory proposal in automated driving stacks. The work highlights the potential of combining spatially-grounded representations with efficient generative models to overcome the limitations of both rule-based systems and computationally expensive diffusion models. The authors note that future work should investigate architectural modifications to handle larger, more complex scenes and specific behaviors like yielding to pedestrians.