Heterogeneous Self-Play for Realistic Highway Traffic Simulation

1. Problem Statement

Realistic highway simulation is essential for the safety evaluation of autonomous vehicles (AVs), particularly for studying rare, safety-critical interactions that are difficult to capture in logged real-world data. Existing approaches face three main limitations:

• Log Replay/Supervised Learning: Preserves realistic behavior but is limited by the data distribution, lacks controllability, and suffers from compounding errors in closed-loop execution.
• Rule-Based Models: Offer controllability but rely on simplified assumptions that fail to capture the richness of multi-agent interactions.
• Urban Self-Play RL: Recent successes in urban driving (e.g., Waymo Open Motion Dataset) do not directly translate to highways due to higher speeds, broader speed ranges, greater density variations, and significant physical heterogeneity (e.g., passenger cars vs. articulated trucks).

The core challenge is to generate a controllable, broadly covering, and behaviorally realistic highway traffic simulator that supports heterogeneous agents without relying on direct imitation of expert logs.

2. Methodology: PHASE Framework

The authors propose PHASE (Policy for Heterogeneous Agent Self-play on Expressway), a context-aware self-play framework.

A. Task Formulation

The problem is modeled as a Conditioned Goal-Reaching Partially Observable Stochastic Game (POSG).

• Agents: A single policy controls heterogeneous agents (passenger cars and articulated tractor-trailers).
• Conditioning: Each agent $i$ is assigned a Cartesian goal $g_i$ and a context vector $C_i = (v_{goal,i}, \alpha_i, T_i, D_i)$, where:
  • $v_{goal,i}$: Target cruising speed (0–40 m/s).
  • $\alpha_i$: Longitudinal control range modulator.
  • $T_i$: Vehicle type (Car/Truck).
  • $D_i$: Vehicle dimensions (including trailer dimensions).
• Objective: Reach the goal region without collision. Speed and yaw alignment are soft objectives handled via reward shaping.

B. Synthetic Scenario Generation

To achieve broad coverage without expert log imitation, PHASE uses an Offline-Online Pipeline:

• Offline: Constructs a reusable pool of start-goal pairs using a lane-graph search. It enumerates reachable endpoints within specific path-distance bounds and limits the number of lane changes ($K$).
• Online: Randomly samples scenarios from the pool, varying agent count, truck proportion, vehicle geometry, and kinematics (initial/goal speeds). This ensures diverse traffic densities and interaction patterns.

C. Simulation Mechanisms for Stability

To stabilize training in high-speed, heterogeneous environments, two mechanisms are introduced:

1. Early Termination: Agents are terminated if they enter an "unrecoverable state" where the goal lies behind their forward vector ($g_i^\top f_i < 0$). This prevents agents from collecting low-value trajectories moving away from the goal.
2. At-Fault Collision Attribution: Unlike standard self-play where collisions penalize all parties, PHASE assigns fault only to the agent physically "behind" the other (based on relative positions and vehicle length). This encourages realistic defensive driving.

D. Reward Design & Curricula

The reward function combines sparse (goal completion, collision) and dense (progress, alignment) terms.

• Coupled Curricula:
  • Reward Curriculum: Gradually shifts focus from dense progress guidance to strict safety/alignment penalties as training progresses.
  • Scenario Curriculum: Increases lane-change ratios and traffic density toward the end of training to improve robustness in congested regimes.
• Training Algorithms:
  • DClamp-PPO: A variant of PPO with a tighter trust region to handle the instability caused by shifting reward landscapes and scenario distributions.
  • Inverse-Agent-Count Reweighting: Balances gradient contributions so that dense worlds (with many agents) do not dominate the training signal over sparse worlds.
  • Action Regularization: Uses a KL-divergence regularizer toward a zero-mean Gaussian prior to encourage smooth, near-zero-centered control (jerk and steering).

E. Model Architecture

PHASE uses an MLP-based architecture enhanced with Cross-Attention and Attention Pooling.

• It encodes ego, partner, and road features separately.
• Cross-attention captures interactions between agents and local topology.
• Conditioning variables are concatenated with ego features before the policy/value heads, allowing a single policy to adapt to different vehicle dynamics and goals.

3. Key Contributions

1. Conditioned Heterogeneous Formulation: A single policy controls diverse vehicle types (cars and trucks) across a wide speed range (0–40 m/s) via joint goal and context conditioning.
2. Scalable Synthetic Pipeline: An offline-online generator creates map-consistent, diverse highway scenes without relying on expert trajectory logs.
3. Stable Self-Play Recipe: A combination of early termination, at-fault attribution, coupled curricula, and robust optimization (DClamp-PPO, reweighting) makes learning-from-scratch feasible for highway regimes.
4. Zero-Shot Transfer & Realism: Demonstrates strong transfer to real-world data and improved behavioral realism over classical baselines.

4. Experimental Results

The model was trained on 22 US highway maps (3 billion steps, ~80 hours on 2x A6000 GPUs) and evaluated on two fronts:

A. Zero-Shot Transfer to Real Data (exiD)

Evaluated on 512 unseen real-world highway scenarios from the exiD dataset:

• Success Rate: PHASE achieved 96.3%, significantly outperforming a prior self-play baseline (Cornelisse et al.) which scored 26.6%.
• Trajectory Error: Reduced Average Displacement Error (ADE) from 6.57m to 2.44m and Final Displacement Error (FDE) from 12.07m to 5.25m.
• Ablation: Removing DClamp, sample reweighting, or curricula led to significant performance drops, validating the stability mechanisms.

B. Distributional Realism Evaluation

Evaluated in a learned trajectory embedding space (using a frozen motion forecasting model) against real highway logs and the Intelligent Driver Model (IDM):

• Fréchet Distance: Improved from 6.35 (IDM) to 5.52 (PHASE).
• Energy Distance: Improved from 0.0233 (IDM) to 0.0186 (PHASE).
• MMD: PHASE consistently showed lower Maximum Mean Discrepancy across various kernel scales, indicating its trajectory distribution aligns more closely with real traffic than IDM.
• Cluster Analysis: PHASE centroids were closer to real trajectory centroids across most behavior clusters, indicating better alignment with real-world driving modes.

5. Significance

This work demonstrates that synthetic self-play can serve as a scalable, controllable route to realistic highway simulation without the need for direct imitation of expert logs. By explicitly conditioning policies on vehicle types and goals, and stabilizing training through novel simulation mechanisms, PHASE bridges the gap between urban self-play research and the distinct demands of high-speed highway driving. The ability to generate rare, safety-critical scenarios on demand while maintaining behavioral fidelity makes it a powerful tool for AV safety validation.