I'm Sorry Driver, I'm Afraid I Can't Do That: Appraising the Safety of LLMs within Automotive Contexts

Technical Summary: Appraising the Safety of LLMs within Automotive Contexts

Problem Statement
The paper addresses the critical safety assurance challenges arising from the integration of Large Language Models (LLMs) into automotive control systems. While LLMs are being rapidly adopted for tasks ranging from infotainment to real-time decision planning and autonomous control, current safety frameworks struggle to accommodate them. The authors identify a "dual challenge" for deployers: they must assure models developed upstream by large AI labs as general-purpose tools (GPAI) within specific, safety-critical downstream vehicle architectures.

The core problem is that existing standards (e.g., ISO 26262, ISO 21448) and emerging guidelines (ISO/PAS 8800) are insufficient for addressing the unique failure modes of LLMs. Specifically, the paper highlights two critical gaps:

1. Value Alignment: The difficulty of ensuring LLM behavior aligns with human intent and safety values, a concept that is nebulous in literature and sparsely covered in standards.
2. Latency: The engineering constraint of real-time response, where LLM inference times may exceed the bounded deadlines required for safe vehicle actuation.

Methodology
The authors propose a structured safety argument pattern based on the Goal Structuring Notation (GSN) to evaluate the safety of LLM-assisted driving. This argument decomposes the top-level claim of safety into specific hazardous events, focusing on:

• Delayed steering due to value misalignment (addressing ISO/PAS 8800 requirements).
• Delayed steering due to latency (addressing ISO 21448 requirements).

To ground this theoretical framework, the paper presents an experimental case study using the open-source repository Talk2Drive. In this setup:

• Architecture: Driver speech is transcribed by Whisper (Speech-to-Text), interpreted by an LLM via scenario-specific prompts, and converted into executable shell commands to actuate the vehicle.
• Models Tested: Four LLMs were benchmarked under identical conditions: GPT-3.5 Turbo, GPT-5.2, Gemini 3 Flash Preview, and Claude Opus 4.5.
• Procedure: The system was tested with 30 standardized voice commands across three scenarios (highway, intersection, parking). Performance was measured via end-to-end latency, transcription quality (Word Error Rate), and "script match" (agreement with a deterministic keyword baseline, not ground truth).

Key Contributions

1. Critical Appraisal: A systematic evaluation of the integration of LLMs into critical automotive systems, distinguishing between upstream (model capability) and downstream (deployment context) assurance challenges.
2. Safety Argument Pattern: A reproducible GSN-based framework for assessing the limitations of current State-of-the-Art LLMs in vehicle architectures, explicitly mapping hazards to existing standards (ISO/PAS 8800 and ISO 21448).
3. Experimental Evaluation: Empirical evidence demonstrating specific hazardous failure modes (latency and alignment) in an end-to-end voice-controlled steering pipeline.

Results
The case study revealed significant barriers to the safe deployment of current LLMs in real-time automotive contexts:

• Latency Issues: End-to-end latency was dominated by the LLM stage. While some models (e.g., GPT-3.5 Turbo) had mean latencies around 1.8 seconds, "reasoning" models (e.g., Gemini 3 Flash Preview) exhibited prohibitive response tails, with maximum latencies exceeding 100 seconds. This renders such models unsuitable for bounded-response actuation required for functional safety.
• Value Alignment Failures: The study observed significant "refusal" behaviors and alignment errors. For instance, when prompted to "Stop here please," a model responded, "Sorry, I cannot assist with that request." This demonstrates a tension between upstream safety guardrails (over-refusal) and downstream functional safety requirements.
• Inconsistency: Script match rates with the baseline ranged from 56.7% to 70.0%, indicating that even "performant" models frequently deviate from expected control policies.
• Argument Incompleteness: The experimental data demonstrated that current solutions for latency (G3) and value alignment (G2) within the proposed safety argument are incomplete. Specifically, existing standards do not adequately cover the mitigation of these specific LLM-induced failure modes.

Significance and Claims
The paper modestly claims to provide a framework for assuring existing proposals rather than proposing a new architecture. Its primary significance lies in:

• Highlighting the Assurance Gap: It explicitly demonstrates that current safety standards and engineering practices are incomplete when applied to general-purpose AI in safety-critical automotive contexts.
• Defining the Scope of Risk: It identifies that issues like latency and value misalignment are not merely theoretical but are observable, operational barriers that prevent the safe integration of LLMs into real-time control loops.
• Future Direction: The authors argue that while LLM integration is being explored technically, the safety community must develop new assurance mechanisms to handle hazardous events related to alignment and latency before these systems can be considered safe for deployment. The paper concludes that the current state-of-the-art is insufficient for real-time control, necessitating further research into robust evaluation and mitigation strategies.