LLMs-Powered Real-Time Fault Injection: An Approach Toward Intelligent Fault Test Cases Generation

1. Problem Statement

Automotive Software Systems (ASSs) are becoming increasingly complex, necessitating rigorous safety validation under the ISO 26262 standard. A critical validation method is Fault Injection (FI), where artificial faults are introduced to verify safety mechanisms. However, current FI methodologies face significant limitations:

• Manual Intensity: Defining fault attributes (type, location, and timing) requires extensive manual effort and domain expertise.
• Scalability Issues: As system complexity grows, the manual process becomes prohibitively expensive and time-consuming.
• Coverage Trade-offs: Practitioners often resort to random or exhaustive fault injection, leading to either insufficient test coverage or an unmanageable number of test cases.
• Lack of Intelligence: Existing methods often lack the ability to systematically generate realistic, requirement-driven fault test cases (TCs) for real-time Hardware-in-the-Loop (HIL) validation.

2. Methodology

The authors propose a novel framework that integrates Large Language Models (LLMs) into the real-time FI workflow to automate the generation of fault TCs from Functional Safety Requirements (FSRs). The approach consists of three main phases:

A. LLM-Assisted Fault TC Generation

1. Input Processing: Natural language FSRs are fed directly into LLMs without complex preprocessing, leveraging the models' inherent natural language processing capabilities.
2. Classification Task: The LLM first classifies FSRs into sensor-related or actuator-related categories. This step reduces uncertainty and focuses the generation process.
3. Fault Attribute Extraction: Based on the classified FSRs, the LLM identifies potential fault locations and types.
   • Output Format: The model outputs a JSON object (key-value pairs) indicating which components (sensors/actuators) should receive a fault injection (value 1) and which should remain healthy (value 0).
   • Concurrency: The framework supports both single faults and concurrent faults (up to two simultaneous faults), adhering to ISO 26262 constraints.
   • Fault Types: The study covers specific fault models including gain, offset, stuck-at, delay, noise, packet loss, drift, and spike, defined mathematically in the paper.
4. Prompting Strategy: The system utilizes few-shot prompting with dynamic context (e.g., lists of supported sensors) to ensure adaptability to changing system configurations without retraining.

B. Real-Time Execution (HIL)

• The generated fault parameters are transmitted to a Hardware-in-the-Loop (HIL) simulation environment.
• A real-time FI framework (using RTI-CANMM) injects faults into the CAN bus signals of a high-fidelity automotive model (gasoline engine and vehicle dynamics) and a physical ECU.
• Crucially, this process treats the System Under Test (SUT) as a black box, requiring no modification to the system architecture.

C. Result Analysis

• System behavior under fault conditions is compared against a "golden run" (fault-free baseline).
• Engineers analyze whether the system's reaction violates FSRs, generating feedback reports for the development phase.

3. Key Contributions

1. Novel Framework: First study to integrate LLMs with real-time FI testing on HIL systems for ASSs.
2. Intelligent Generation: A systematic approach to derive representative fault TCs directly from textual FSRs, optimizing the testing process.
3. Classification & Generation Pipeline: An LLM-based workflow that classifies requirements (sensor vs. actuator) and generates precise fault injection vectors (single and concurrent).
4. Comprehensive Evaluation: A rigorous comparison of multiple LLMs (both proprietary and open-source) regarding data size, prompt engineering, and generation accuracy.
5. Real-World Validation: Demonstration of the framework's efficacy on high-fidelity automotive models, analyzing system robustness under injected faults.

4. Experimental Results

The study evaluated several models: gpt-4o, gpt-4o-mini, phi-4-14b, llama-3-70b, and qwen2.5-7b.

• FSR Classification:

  • gpt-4o achieved the highest performance with an F1-score of 88.0% for classifying sensor vs. actuator requirements.
  • Open-source models like qwen2.5-7b and phi-4-14b showed strong performance (85.7% and 83.2% respectively) with sufficient few-shot examples.

• Fault TC Generation:

  • gpt-4o demonstrated superior performance with an F1-score of 97.5% (at N=8 examples) for generating fault TCs.
  • phi-4-14b emerged as the best open-source alternative, achieving an F1-score of 96.6%, showing high robustness in structured generation tasks.
  • Batch Processing: The study analyzed batch generation efficiency. While gpt-4o maintained high accuracy, phi-4-14b struggled with large context windows (dropping to 0% F1 when exceeding context limits), highlighting the importance of prompt management.

• Negative Constraint Handling:

  • Models were tested on their ability to avoid injecting faults into unsupported sensors. phi-4-14b excelled here with 97.7% accuracy, proving its ability to adhere to strict constraints.

• Real-Time Impact:

  • Execution on the HIL system confirmed that injected faults (e.g., delay in APP sensor, stuck-at in RPM sensor) successfully triggered FSR violations and observable system degradation (e.g., engine speed deviation, torque spikes, temperature instability), validating the realism of the generated test cases.

5. Significance and Future Work

• Efficiency & Cost: The approach significantly reduces the manual effort, time, and cost associated with creating fault test cases for complex automotive systems.
• Safety Enhancement: By systematically generating realistic fault scenarios based on requirements, the method improves test coverage and the reliability of safety mechanisms.
• Flexibility: The use of few-shot prompting allows the system to adapt to new sensors or requirements dynamically without model retraining.
• Future Directions: The authors plan to expand the framework to include communication-level and ECU-level faults, and to investigate using LLMs to provide rationales for their predictions to aid decision-making.

In conclusion, this paper establishes a viable path toward intelligent, automated safety validation for autonomous and connected vehicles, leveraging the generative capabilities of LLMs to bridge the gap between high-level safety requirements and low-level real-time fault injection.