Large Language Model based Interactive Decision-Making for Autonomous Driving

Technical Summary: Large Language Model-based Interactive Decision-Making for Autonomous Driving

1. Problem Statement

Current autonomous driving (AD) systems struggle in high-conflict, mixed-traffic scenarios (where human-driven vehicles (HDVs) and autonomous vehicles (CAVs) coexist). Most existing systems suffer from three primary issues:

• Over-conservatism: To ensure safety, CAVs often adopt overly cautious behaviors (e.g., stopping at intersections), leading to traffic deadlocks and reduced efficiency.
• Lack of Proactive Interaction: Traditional rule-based, game-theoretic, or machine learning models often fail to anticipate or negotiate the implicit intents of human drivers.
• Fragmented Communication: Existing external Human-Machine Interfaces (eHMI) are typically signal-centric (e.g., simple lights) and are not tightly coupled with the vehicle's internal reasoning, leading to a lack of transparency and trust.

2. Methodology

The authors propose a closed-loop framework that integrates semantic scene understanding, LLM-driven reasoning, and natural language interaction. The pipeline consists of four integrated stages:

• A. Semantic Scene Modeling (Object-Process Methodology - OPM):
  To avoid overwhelming the LLM with raw numerical data, the system uses OPM to abstract low-level perceptual data into a structured semantic graph. This graph consists of:

  • Objects: Traffic participants (vehicles, pedestrians).
  • Processes: Operational states (lane changing, accelerating).
  • Relations: Interaction dynamics (potential collisions, right-of-way conflicts).
    This abstraction reduces perceptual noise and aligns the input with the LLM’s linguistic reasoning capabilities.

• B. LLM-Driven Interactive Decision-Making:
  The framework uses a Large Language Model (specifically Llama3-7B) to perform two critical tasks:

  • Intent Parsing: The LLM parses both explicit intents (turn signals, voice) and implicit intents (inferred from historical trajectories and driving styles like "aggressive" or "conservative") of surrounding agents.
  • Decision Correction: The system first generates a baseline decision using a Multinomial Logit Model (based on utility functions for safety, efficiency, and comfort). The LLM then acts as a "semantic reasoning module" to refine this decision, ensuring temporal consistency and social rationality.

• C. Trajectory Optimization:
  Once a high-level maneuver (e.g., "left turn") is selected, the system employs Monte Carlo sampling to generate candidate trajectories using fifth-order polynomials. These candidates are evaluated against safety and efficiency constraints to select the optimal executable path.

• D. Language-Based eHMI Interaction:
  The final decision is translated by the LLM into concise, natural-language messages (e.g., "I am slowing down; please go ahead") and broadcast via an external HMI. This creates a bidirectional communication loop between the CAV and human road users.

3. Key Contributions

1. OPM-based Abstraction: A novel method to transform complex spatiotemporal data into a structured semantic format that enhances LLM reasoning efficiency and accuracy.
2. Integrated Decision-Interaction Framework: A unified architecture that links perception, intent-aware decision-making, and natural language communication.
3. Human-like Interaction Mechanism: The use of LLMs to bridge the gap between machine logic and human social expectations through natural language eHMI.

4. Experimental Results

The researchers validated the model using the Tongji University Cluster Driving Simulator across intersection and merging scenarios.

• Performance Metrics: Compared to traditional IDM (Rule-based) and GT (Game-Theoretic) models:
  • Efficiency: The proposed model achieved higher average speeds across low, medium, and high-speed scenarios.
  • Safety: It significantly reduced "Average Conflict Duration" (by ~35% compared to IDM and ~10% compared to GT), indicating faster and safer conflict resolution.
  • Comfort: It maintained a competitive "Average Jerk" profile, avoiding the abrupt movements seen in rule-based models.
• OPM Validation: Experiments showed that OPM-structured input improved LLM accuracy by 12.2% and reduced inference latency from 6.59s to 3.88s compared to raw data input.
• Turing Test: A double-blind study with 22 volunteers showed that human drivers could not reliably distinguish between the LLM-controlled vehicle and a human driver (accuracy rate of 0.57, near random chance), and rated the model's behavior as highly "natural."

5. Significance

This work demonstrates that coupling semantic scene abstraction with the reasoning power of LLMs provides a practical pathway toward "trustworthy" autonomous driving. By moving beyond simple obstacle avoidance toward socially-aware negotiation, the proposed method addresses the critical bottleneck of public acceptance in dense, mixed-traffic urban environments.