Automated Semantic Fault Localization in SysML v2: A Human-in-the-Loop Framework Using Knowledge-Graph Augmented LLMs

Technical Summary: Automated Semantic Fault Localization in SysML v2

Problem Statement
Model-Based Systems Engineering (MBSE) using SysML v2 offers formal validation for syntactic rules, yet it struggles to detect semantic faults. These are errors where a model is syntactically valid and passes compiler checks but violates domain-specific engineering constraints, such as physically incompatible connections (e.g., linking a fluid interface to an electrical port) or dimensional inconsistencies. Current Large Language Models (LLMs) applied to SysML v2 face significant hurdles: a scarcity of high-quality training data, a tendency to hallucinate syntax or non-existent constructs, and an inability to reason over large systems without explicit domain context. Furthermore, existing approaches often rely on autonomous generation, which lacks the necessary engineering oversight for safety-critical systems.

Methodology
The authors propose a human-in-the-loop framework that shifts the LLM's role from autonomous model generation to automated fault localization and repair suggestion. The methodology consists of four primary components:

1. Domain Knowledge Graph (KG): A KG was constructed to encode physical compatibility rules and engineering constraints for the vehicle systems domain. This graph maps system interfaces to physical domains (e.g., mechanical, electrical, fluid) and International System of Quantities (ISQ) types to valid units. The KG serves two functions: it guides the generation of synthetic training data by defining valid mutation rules, and it augments inference prompts to ground repair suggestions in explicit engineering constraints.

2. Synthetic Dataset Construction: To address data scarcity, the authors synthesized a dataset of 8,301 entries starting from a base corpus of 256 valid SysML v2 models.

   • Syntax Faults: A mutation engine applied 12 heuristic operators (e.g., breaking imports, unbalancing brackets) to generate 5,497 syntactically invalid samples.
   • Semantic Faults: Using the KG, the engine applied 5 semantic mutation heuristics (e.g., port mismatches, unit swaps) to create 1,402 samples that are syntactically valid but semantically incorrect.
   • Correct Examples: 1,402 valid samples were included to allow the model to distinguish between correct and faulty code.
     The dataset was split into training (70%), validation (15%), and test (15%) sets, ensuring no data leakage between source models and mutated samples.

3. Patch-Based Output: Instead of generating full code rewrites, the models were fine-tuned to output unified diff patches. This format specifies changes via added and removed lines with context, reducing token length and facilitating human review.

4. Supervised Fine-Tuning (SFT): Two Small Language Models (SLMs)—Qwen-2.5 Coder 1.5B and DeepSeek Coder 6.7B—were fine-tuned using Low-Rank Adaptation (LoRA). The models were trained to predict a status indicator (correct/incorrect) followed by a repair patch, conditioned on the input code and relevant domain rules from the KG.

Key Contributions

• Dataset and Synthesis Methodology: The creation of a labeled dataset containing syntactic and semantic faults, synthesized via a KG-guided mutation engine to overcome the scarcity of SysML v2 training data.
• Domain-Aware Pipeline: A fault localization pipeline that integrates a knowledge graph to guide both data generation and inference-time prompt augmentation, enabling the detection of latent semantic violations.
• Evaluation of SLMs: An empirical evaluation demonstrating that fine-tuned SLMs can effectively identify and repair both syntax and semantic errors in SysML v2, outputting results as unified diff patches.

Results
Evaluation on a held-out test set of 1,184 samples yielded the following findings:

• Performance Improvement: Fine-tuning dramatically improved semantic fault repair accuracy. Baseline pre-trained models achieved less than 3% accuracy on semantic errors (ranging from 0.62% to 2.47%), whereas fine-tuned models achieved over 91% (91.4%–95.7%).
• Syntax vs. Semantics: Fine-tuning also significantly improved syntax repair, with pass@1 scores rising from ~17–36% (baseline) to ~89–98% (fine-tuned).
• Efficiency: Models fine-tuned to output patches reduced the average token length by approximately 50% compared to models generating full code rewrites, and by up to 75% compared to baseline models.
• Impact of Domain Rules: While providing domain rules in the prompt offered marginal gains for baseline models, the combination of fine-tuning and KG-augmented prompts was critical for high semantic repair performance.

Significance and Claims
The paper claims that this framework offers a practical path toward AI-assisted model verification that complements existing MBSE tools. By positioning the LLM as a domain-aware assistant rather than an autonomous agent, the approach preserves human judgment in the design process while extending verification capabilities beyond syntax to domain physics. The authors assert that their heuristic-based data augmentation methodology provides a viable solution to the training data scarcity problem, enabling small language models to detect domain-level errors that standard compilers miss. The work emphasizes that explicit, well-structured domain knowledge is essential for applying LLMs to SysML v2, as general-purpose models lack the necessary exposure to valid syntax and engineering constraints.