Technical Summary: TrafficAlign

Problem Statement

While large language models (LLMs) have been explored for generating traffic scenarios to test autonomous driving (AD) systems, current approaches face significant limitations. Pretrained LLMs often lack sufficient domain-specific knowledge, leading to the generation of homogeneous scenarios that fail to align with real-world traffic distributions, particularly in specific geographic contexts (e.g., large cities vs. mountainous regions). Existing methods relying on manual natural language descriptions are labor-intensive and do not scale, while automated approaches like ChatScene, which prompt LLMs directly, struggle to capture fine-grained, location-specific traffic characteristics. Consequently, there is a need for an automated framework that can synthesize realistic, region-specific traffic scenarios and align LLMs with actual real-world traffic distributions to improve both the testing and training of AD models.

Methodology

The paper proposes TrafficAlign, an automated framework designed to synthesize, validate, and align LLMs with real-world traffic scenarios derived from driving videos. The pipeline consists of three primary stages:

1. Data Synthesis

TrafficAlign utilizes real-world driving videos as the primary data source to capture the underlying traffic distributions of target regions.

• Video Collection: The system samples frames from diverse driving videos (e.g., from YouTube) covering various geographic regions, including metropolitan areas, national parks, small towns, and international locations.
• Scenario Extraction: A multimodal LLM (specifically GPT-4.1 nano) is employed to analyze selected video frames. Using a compositional prompt strategy (incorporating role-playing, chain-of-thought reasoning, and few-shot learning), the LLM generates natural language descriptions of the traffic scenario. These descriptions cover the road network, environmental context (weather, time), and the behavior/position of all actors (ego vehicle, NPCs, pedestrians).

2. Data Validation

To ensure the quality and validity of the synthesized data, TrafficAlign employs a rigorous validation loop:

• DSL Translation: Natural language descriptions are translated into a formal, domain-specific language (DSL) representation. This conversion is performed by an LLM (GPT-5) acting as a traffic expert, guided by a specific grammar.
• Syntactic and Semantic Checking: The system performs syntactic checks to ensure the DSL is well-formed. It then conducts semantic validation to verify that essential attributes (e.g., time of day, weather, actor behaviors) are present.
• Self-Refinement: If a scenario is identified as semantically incomplete but recoverable, the system appends diagnostic messages to the prompt and requests the scenario extractor to regenerate the description. Irrelevant frames (e.g., titles or screens) are discarded.

3. LLM Alignment

Once validated, the synthesized scenario descriptions are used to align the LLM with real-world traffic distributions.

• Fine-Tuning: The framework performs standard supervised fine-tuning (SFT) on a generator model (Llama-3.2-3B-Instruct) using the validated scenario descriptions.
• Parameter-Efficient Tuning: The process utilizes Low-Rank Adaptation (LoRA) applied to attention and MLP layers to efficiently adapt the model.
• Output: The resulting aligned LLM can be prompted to generate traffic scenario descriptions that are statistically consistent with the real-world traffic distributions of the target region.

Key Contributions

1. Framework Proposal: The introduction of TrafficAlign, an automated pipeline that synthesizes traffic scenarios from real-world videos, validates them via DSL translation, and aligns LLMs with these distributions.
2. Testing Effectiveness: Demonstration that scenarios generated by TrafficAlign are significantly more effective at stress-testing autonomous driving models than state-of-the-art baselines.
3. Training Improvement: Evidence that fine-tuning driving models on TrafficAlign-generated scenarios significantly enhances their safety and robustness.
4. Regional Alignment: A qualitative study showing that the framework successfully captures and replicates traffic distributions across geographically diverse regions.

Experimental Results

The authors evaluated TrafficAlign against five baselines (including ChatScene, adversary-based methods, and rule-based methods) and five unaligned LLM baselines using three autonomous driving models (PPO, SAC, TD3) within the SafeBench platform and CARLA simulator.

• Testing Effectiveness: TrafficAlign-generated scenarios revealed up to 10.8% more collisions on average across the three driving models compared to the strongest baseline (ChatScene). In terms of the overall score, TrafficAlign induced a relative reduction of 5.0% to 35.6%, indicating it generates more challenging and safety-critical scenarios.
• Training Improvement: When driving models were fine-tuned on TrafficAlign-generated scenarios, the collision rate decreased by 36.1% compared to the original, pre-fine-tuned models. This improvement was superior to fine-tuning with scenarios from the ChatScene baseline, which showed a smaller reduction in collision rates.
• Distribution Alignment: Qualitative analysis using UMAP visualization of scenario embeddings demonstrated that TrafficAlign-generated scenarios co-locate and overlap significantly with real-world traffic clusters. In contrast, unaligned LLM baselines formed distinct, displaced clusters with minimal overlap with real-world data.

Significance and Claims

The paper claims that TrafficAlign addresses the critical gap between general LLM knowledge and specific, real-world traffic distributions. By automating the extraction of scenarios from video and aligning LLMs through a validation loop, the framework enables:

• More Rigorous Testing: The ability to generate diverse, region-specific scenarios that expose safety vulnerabilities in AD models more effectively than current methods.
• Targeted Model Improvement: The capacity to use these realistic scenarios to fine-tune AD models, resulting in substantial reductions in collision rates and improved overall driving performance.
• Scalability: A move away from labor-intensive manual scenario design toward an automated, data-driven approach that can adapt to various geographic contexts.

The authors conclude that TrafficAlign facilitates the automatic construction of traffic-safety scenario benchmarks, offering a pathway for consistent testing and targeted improvement of autonomous driving systems.