💻 computer science

Traffic Scene Generation from Natural Language Description for Autonomous Vehicles with Large Language Model

This paper proposes TTSG, a modular framework that leverages Large Language Models within a tightly controlled pipeline to generate realistic, controllable, and safety-oriented traffic scenes from natural language descriptions, achieving superior performance in collision reduction and driving action reasoning compared to existing methods.

Original authors: Bo-Kai Ruan, Hao-Tang Tsui, Yung-Hui Li, Hong-Han Shuai

Published 2026-03-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bo-Kai Ruan, Hao-Tang Tsui, Yung-Hui Li, Hong-Han Shuai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a director of a massive, high-tech movie studio for self-driving cars. Your job is to test these cars by throwing every possible traffic nightmare at them: sudden pedestrians, angry drivers cutting you off, rainy nights, and confusing intersections.

In the past, making these "movies" (simulations) was like trying to build a set by hand. You had to manually place every car, draw every road, and program every traffic light. It was slow, rigid, and you couldn't easily ask for a specific, weird scenario like, "What if a fire truck tries to merge while a cyclist swerves in front of a stop sign?"

This paper introduces a new way to make these movies using a "Magic Scriptwriter" (a Large Language Model, or LLM).

Here is how the paper's system, called TTSG, works, explained through a simple analogy:

1. The Magic Scriptwriter (The LLM)

Instead of a human engineer typing code, you just talk to the computer in plain English.

You say: "I want a rainy night scene where a pedestrian crosses without a crosswalk, and a car is speeding toward them."
The LLM acts as a translator: It doesn't just hear the words; it breaks them down into a technical checklist. It understands that "rainy night" means weather = wet, "pedestrian" means agent type = human, and "no crosswalk" means object = none.

2. The Infinite Library (The Road Graph)

The system has a giant digital library of every road in the city (a "Road Graph").

The Problem: If you ask for a road with a stop sign and a crosswalk, the system can't just guess where to put it. It has to find a real road in the database that actually has those features.
The Solution: The LLM acts like a librarian. It scans the library and pulls out a list of roads that might fit your description (e.g., "Road A has a stop sign, Road B has a crosswalk").

3. The Smart Matchmaker (The Ranking Algorithm)

This is the paper's secret sauce. Just because a road has a stop sign doesn't mean it's the right place for your specific scene.

The Analogy: Imagine you are trying to fit a square peg into a round hole. You have a list of candidate roads, but you also have a plan for where the cars and pedestrians need to stand.
The Magic: The system runs a "compatibility test." It asks: "If I put the pedestrian here and the car there, does the road geometry actually allow them to move without crashing immediately?"
It scores every road. The road that fits your story perfectly gets the highest score and is chosen. This ensures the scene makes sense physically, not just linguistically.

4. The Director's Chair (Scene Generation)

Once the LLM has picked the perfect road and planned exactly where every actor (car, pedestrian, truck) should stand and what they should do, it hands the script to the simulator (CARLA).

The simulator builds the scene instantly.
The result is a realistic video of the traffic scenario, ready to be used to train self-driving cars.

Why is this a Big Deal?

1. No More "Pre-Set" Scenes
Before this, if you wanted to test a specific danger, you had to find a real video of it or build it from scratch. Now, you can invent any scenario on the fly. It's like having a video game where you can type "create a zombie apocalypse in a school zone" and the game builds it instantly.

2. Safety First
The paper tested this by creating "dangerous" scenarios (like cars cutting each other off). They found that self-driving cars trained on these AI-generated scenes became much safer. They crashed 3.5% of the time in tests, which was the lowest rate compared to other methods. It's like giving a driver's ed student a simulator that generates thousands of unique, tricky situations they've never seen before, so they are ready for anything on the real road.

3. Teaching the AI to "Think"
The authors also used these generated scenes to teach other AI models how to explain what is happening.

Before: An AI might see a car turning left and just say, "Car turning."
After: Because it was trained on these detailed, logical scenes, the AI can say, "Car turning left slowly to give way to an oncoming truck." It learned the reasoning, not just the action.

In a Nutshell

This paper turns the complex, technical job of building traffic simulations into a simple conversation. You describe the chaos you want to see, and the AI acts as the architect, the set designer, and the safety inspector all at once, building a perfect, safe, and realistic traffic world from thin air.

1. Problem Statement

The development and safety evaluation of autonomous driving (AD) systems rely heavily on realistic traffic scene datasets. While real-world data (e.g., nuScenes, Waymo) is valuable, it suffers from:

Limited Controllability: It is difficult to systematically generate specific failure modes, edge cases, or high-risk scenarios (e.g., aggressive lane changes, near-collisions).
Safety Risks: Collecting dangerous scenarios in the real world is unsafe.
Limitations of Existing Simulators: Current tools like CARLA or MetaDrive often rely on random sampling (lacking intentional control) or replaying logged trajectories (lacking novelty).
Gaps in Instruction-Driven Generation: Previous methods using Large Language Models (LLMs) often require structured inputs, predefined spawn points, or fixed map routes. They struggle to interpret free-form natural language, handle multi-agent coordination, or incorporate environmental conditions (weather, signals) without rigid constraints.

Core Challenge: How to generate spatially valid, semantically coherent, and controllable traffic scenes directly from unstructured natural language descriptions without predefined routes or spawn points.

2. Methodology: The TTSG Framework

The authors propose TTSG (Text-to-Traffic Scene Generation), a training-free, modular framework that integrates LLMs into a tightly controlled pipeline. The process consists of five stages:

A. Infrastructure Construction

Road Graph ( $G$ ): Built-in CARLA maps are converted to OpenDRIVE format. Road features (signals, static objects, junctions, lane configs) are parsed into a graph where edges represent connectivity. This allows for efficient querying of road attributes.
Agent Set: Agents are categorized by type (e.g., ambulance, pedestrian, truck) and behavior (cautious, aggressive) to support flexible instantiation.

B. The Five-Stage Pipeline

Prompt Analysis:
- An LLM decomposes the user's free-form text into explicit, structured components: required signals, objects, weather, and agent configurations (type, relative position, action).
- Innovation: Instead of using token-heavy Chain-of-Thought (CoT) reasoning for the whole process, the authors use a specific decomposition step to reduce token usage while maintaining accuracy.
Road Candidate Retrieval:
- Based on the parsed conditions, the LLM generates filtering criteria (e.g., "min 2 lanes," "no traffic lights," "must have crosswalk").
- These criteria are applied to the Road Graph to retrieve a set of candidate roads ( $R_c$ ).
Agent Planning:
- The LLM generates detailed plans for all agents (including the ego vehicle).
- It specifies agent types, actions (e.g., "turn left," "block ego"), relative positions (e.g., "front right," "on shoulder"), and inter-agent relationships.
- The framework supports complex multi-agent coordination and relative positioning beyond just the ego vehicle.
Plan-Aware Road Ranking (Novelty):
- To ensure spatial validity, a ranking algorithm scores candidate roads based on their compatibility with the planned agent maneuvers.
- Scoring Logic: A road receives a score increment for every agent condition it satisfies (e.g., does the road have a left-turn lane if an agent is planned to turn left? Is there enough space?).
- Selection: The road with the highest score ( $r^*$ ) is selected. If tied, one is chosen randomly to ensure diversity. This step eliminates the need for predefined spawn points by dynamically selecting the optimal road layout.
Scene Generation:
- The selected road and agent plans are rendered into an executable simulation (using CARLA).
- The system supports closed-loop interactions (e.g., blocking agents adapting to the ego vehicle) and sequential events (iterative planning for multi-stage scenarios).

3. Key Contributions

Free-Form to Scene Pipeline: A novel framework that generates traffic scenes directly from unstructured natural language, eliminating the need for predefined routes, spawn points, or rigid prompt formats.
Plan-Aware Road Ranking: A unique algorithm that jointly evaluates road geometry and agent plans to ensure semantic coherence and spatial feasibility, solving the "hallucination" problem where LLMs might plan actions impossible on the selected road.
Comprehensive Scenario Support: The system handles routine traffic, safety-critical scenarios (e.g., sudden swerves), and conditional generation (weather, signals) as well as sequential multi-stage events.
Dual Utility: The framework serves both as a training data generator (improving agent safety) and a data source for captioning models (improving action reasoning).

4. Experimental Results

The framework was evaluated using the CARLA simulator and the SafeBench benchmark.

Safety Performance (Collision Rate):
- Trained on TTSG-generated scenes, autonomous agents achieved the lowest average collision rate (3.5%) across three critical scenarios (Straight Obstacle, Lane Changing, Unprotected Left Turn) compared to state-of-the-art methods like ChatScene (8.0%), AdvSim (27.0%), and Learning-to-Collide (21.0%).
Captioning and Reasoning:
- Driving captioning models (e.g., ADAPT) trained on TTSG data showed a significant improvement in reasoning capabilities.
- CIDEr scores increased by over 30 points for action reasoning in critical scenarios, indicating the model learned to better narrate why an agent took a specific action (e.g., "slowing down to yield").
Ablation Studies:
- Plan Quality: The proposed analysis-based approach achieved high Agent Accuracy (AA) and Road Accuracy (RA), comparable to CoT methods but with significantly fewer tokens.
- Road Ranking: Removing the ranking strategy caused a drop in Scene Accuracy (SA) from 0.80 to 0.56, proving its necessity for spatial validity.
- Model Agnostic: The pipeline performed robustly across different LLMs, from lightweight open-source models (Gemma3-12B) to powerful API models (GPT-4o, Claude-Sonnet-3.5).
Diversity: The system demonstrated high Road Diversity (RD) and Agent Diversity (AD), successfully generating unique scenes for the same prompt without relying on fixed selections.

5. Significance and Impact

Safety-Oriented Simulation: TTSG provides a scalable, safe, and controllable way to generate high-risk scenarios essential for stress-testing autonomous driving systems, addressing the "long-tail" problem in AD development.
Interpretability: By using LLMs as planners with explicit constraints and ranking, the generation process is more interpretable than black-box diffusion models or random sampling.
Efficiency: The framework is training-free (no fine-tuning of the LLM required) and computationally efficient due to its structured decomposition strategy.
Future Directions: The authors plan to extend the system to generate entirely new traffic objects and leverage LLMs for direct multi-agent action control, further bridging the gap between natural language intent and complex traffic dynamics.

In conclusion, TTSG represents a significant step forward in making autonomous driving simulation more flexible, safe, and accessible through natural language interaction, effectively turning text descriptions into rigorous safety benchmarks.