Fault Detection and Explainable Classification in Automotive HIL Validation via Denoising Autoencoders and In-Context Large Language Models

Technical Summary: Fault Detection and Explainable Classification in Automotive HIL Validation

Problem Statement
The validation of modern Automotive Software Systems (ASSs) generates massive, multivariate test recordings from Hardware-in-the-Loop (HIL) simulations. Current validation workflows rely heavily on effort-intensive manual reviews and rule-based evaluations, which struggle to detect faults that fall outside predefined rules or historical data. While machine learning (ML) and deep learning (DL) have advanced fault diagnosis, supervised models require large labeled datasets, generalize poorly to unseen conditions, and lack interpretability. Conversely, unsupervised methods often lack the reasoning capabilities to classify specific fault types or provide root-cause explanations. There is a critical gap in frameworks that can simultaneously detect anomalies without fault labels, classify them with high generalizability across different powertrains and driving regimes, and provide interpretable diagnostic reports for engineers.

Methodology
The authors propose a generalisable, two-phase framework that decouples fault detection from fault classification, utilizing Denoising Autoencoders (DAEs) and frozen Large Language Models (LLMs).

1. Phase 1: Label-Free Fault Detection (DAE)

   • A DAE is trained exclusively on healthy signal data. By adding Gaussian noise during training, the model learns to reconstruct the intrinsic structure of normal behavior.
   • During validation, the reconstruction error is analyzed. Windows with errors exceeding a predefined threshold are flagged as abnormal. This approach eliminates the need for fault labels during the detection phase.
   • The DAE is trained on data from two distinct powertrains (gasoline-engine and electric-vehicle) and validated across three driving regimes (urban, non-urban, highway).

2. Phase 2: Explainable Classification (Frozen LLM)

   • Abnormal windows are not fed directly as raw time-series data. Instead, they are converted into compact, structured textual evidence. This evidence includes statistical descriptors (e.g., z-score, median, max deviation) relative to a time-aligned healthy reference.
   • Crucially, the evidence representation separates direct root-signal deviations from propagated effects to encourage root-cause reasoning.
   • A frozen LLM (no parameter updates) classifies the fault using in-context learning. The system employs both zero-shot (instruction only) and few-shot (instruction + labeled examples from a leakage-safe temporal region) prompting strategies.
   • The model outputs a constrained JSON object containing the predicted class, ranked alternatives, confidence value, fault location, and a short, evidence-based explanation.

Key Contributions

• Two-Phase Framework: A novel architecture that separates label-free anomaly detection (via DAE) from interpretable classification (via frozen LLM), reducing runtime overhead while retaining detailed analysis.
• First Application to HIL Data: This is the first study applying LLMs to digital test-drive datasets generated from real-time HIL simulations, specifically addressing faults propagating through physically and functionally coupled signals.
• Structured Evidence Representation: An input format that distinguishes between direct root causes and propagated symptoms, constraining the LLM to a structured output for traceable justification.
• Systematic Evaluation: A comprehensive evaluation of eight open-source LLMs (ranging from 7B to 70B parameters) under zero-shot and few-shot conditions, quantifying the impact of model scale, input representation, and prompting strategy.

Results

• Fault Detection: The DAE achieved robust performance across diverse conditions, with an average F1-score of 0.9724 across powertrains and 0.9844 across driving regimes. The mean reconstruction error remained below 0.03, demonstrating high generalizability.
• Classification Performance:
  • Zero-Shot: Performance was insufficient for practical use, with the best model (Gemma 2 9B) achieving only a 0.519 F1-score.
  • Few-Shot: Prompting with examples dramatically improved results. Under stable regimes (highway/urban), few-shot prompting achieved perfect discrimination (F1 = 1.0) for several models.
  • Model Scale vs. Strategy: A critical finding is that prompting strategy governs classification quality more than parameter count. A 9-billion-parameter model (Gemma 2 9B) with few-shot prompting outperformed every zero-shot medium and large model.
• Model Selection: Mistral Small 24B was selected as the primary pipeline model. It offered the best balance of accuracy, class-balanced reliability (MCC), calibration (low Brier score/ECE), and inference cost (approx. 4.9s per window).
• Interpretability: The generated explanations successfully linked predictions to specific physical fault locations and signal evidence, allowing engineers to verify diagnoses on technical grounds.

Significance and Claims
The paper claims that this framework provides a scalable, interpretable solution for automotive HIL validation that does not require task-specific fine-tuning or extensive labeled datasets. By decoupling detection from classification and leveraging in-context learning, the approach allows engineers to adapt to new fault sets with minimal cost. The study emphasizes that for safety-critical applications, the ability to provide traceable, evidence-based explanations is as vital as detection accuracy.

Limitations and Future Work
The authors modestly note that the evaluation is currently limited to HIL simulation data; performance on real-world road-test data remains to be verified. The taxonomy is restricted to ten single-fault classes, and the system relies on leakage-safe few-shot exemplars, as zero-shot performance is significantly lower. Additionally, the 3–7 second inference time is currently more suitable for offline analysis than real-time online deployment. Future work aims to validate the framework on real-world data, extend it to simultaneous and interacting faults, and explore techniques like LoRA and model distillation to improve efficiency.