Exploring the Potential of Large Language Models in Simulink-Stateflow Mutant Generation

1. Problem Statement

Mutation analysis is a critical technique for assessing test suite adequacy in safety-critical Cyber-Physical Systems (CPS). However, applying this to Simulink-Stateflow models presents unique challenges compared to traditional source code:

• Complexity: These models integrate continuous dynamics with discrete-event behaviors, featuring hierarchical structures (states, transitions, junctions).
• Limitations of Conventional Methods: Rule-based mutation operators often generate redundant, equivalent (semantically identical to the original), or non-executable mutants.
• Limitations of Existing ML Approaches: Previous machine learning attempts are constrained by limited training data and scalability, often producing unrealistic or syntactically invalid mutants.
• The Gap: There is a lack of systematic evaluation on whether Large Language Models (LLMs), with their ability to reason over semantic and structural patterns, can generate high-quality, domain-specific mutants for Simulink-Stateflow models that are both syntactically valid and behaviorally diverse.

2. Methodology

The authors propose an automated pipeline to generate mutants using LLMs. The process involves four main stages:

1. Model Parsing: The Simulink-Stateflow model is converted into a structured JSON representation containing states, transitions, and junctions.
2. Prompt Engineering: A prompt generator creates inputs for the LLM based on a standardized template containing:
   • Context: The JSON model data.
   • Input Data: Specific elements to mutate (for local strategies) or the whole model (for global strategies).
   • Few-Shot Examples: Real mutation examples from a baseline algorithm to guide the LLM.
   • Output Requirements: Strict JSON formatting rules and constraints (e.g., do not mutate names/positions).
3. LLM Inference: The prompt is sent to an LLM (via OpenAI API or local Ollama deployment), which generates a JSON response containing proposed mutations.
4. Mutant Reconstruction: A parser converts the LLM's JSON output back into executable Simulink models.

Experimental Design:

• Models: Evaluated 8 LLMs (3 closed-source GPT family, 5 open-source: Mistral, DeepSeek, Llama 3, Gemma, Hermes).
• Case Studies: 4 diverse Simulink-Stateflow models (Door controller, Fridge controller, Elevator, Pacemaker).
• Variables:
  • Mutation Strategies: Global (entire model access) vs. Local (specific element focus).
  • Prompt Strategies: Zero-shot vs. Few-shot (varying number of examples).
  • Temperature: 6 values (0.2 to 1.0) to test creativity vs. determinism.
• Scale: A total of 38,400 LLM-generated mutants were produced and compared against a baseline of 800 rule-based mutants.
• Metrics: Effectiveness (Generability, Compilability, Duplication Rate, Equivalent Rate, Mutant Quality) and Efficiency (Time, Cost).

3. Key Contributions

1. Systematic Evaluation: The first comprehensive study evaluating 8 state-of-the-art LLMs for Simulink-Stateflow mutant generation across multiple strategies and configurations.
2. Automated Pipeline: Development of an open-source tool that bridges Simulink models and LLMs via JSON intermediaries.
3. Strategic Insights: Identification of optimal configurations:
   • Global Mutation + Few-Shot Prompting yields the highest syntactic validity (generability/compilability).
   • Local Mutation significantly reduces duplicate and equivalent mutants, improving diversity.
   • Low-to-Medium Temperature (0.2–0.6) produces the most reliable and high-quality mutants.
4. Error Analysis: Identification of four primary failure modes in LLM generation: invalid references, incorrect transition sources/destinations, undefined variables, and syntax errors.
5. Open Resources: Release of the prototype tool and the complete dataset of 38,400 mutants on GitHub to facilitate reproducibility.

4. Key Results

• Efficiency: LLMs generated mutants up to 13x faster than the manually engineered baseline (e.g., GPT-3.5 took ~0.5s vs. baseline ~6.6s).
• Quality & Diversity:
  • LLMs produced significantly fewer equivalent mutants (e.g., Gemma 2 had ~6x fewer equivalents than the baseline in the Door model).
  • Duplication rates were drastically reduced (e.g., from 0.67 to 0.31 in the Door model).
  • Overall mutant quality (ability to be detected by specific test cases) was higher for LLMs compared to the baseline.
• Validity Trade-offs: While LLMs had slightly lower generability and compilability rates (approx. 90–97%) compared to the baseline (near 100%), the mutants they did produce were of higher behavioral value.
• Strategy Impact:
  • Global Strategy: Better for syntactic correctness and speed.
  • Local Strategy: Better for reducing redundancy and improving semantic diversity.
• Prompting: Few-shot prompting consistently outperformed zero-shot approaches. However, increasing the number of examples beyond a certain point yielded diminishing returns and slightly increased duplication.

5. Significance

This research demonstrates that LLMs are a viable and superior alternative to traditional rule-based mutation operators for complex modeling environments like Simulink-Stateflow.

• Practical Impact: By generating fewer redundant and equivalent mutants, LLMs reduce the computational cost of mutation testing (a major bottleneck in CPS development) while increasing the likelihood of uncovering subtle logic errors.
• Domain Advancement: It bridges the gap between general-purpose code generation and domain-specific modeling, proving that LLMs can understand the hierarchical and semantic constraints of Stateflow diagrams.
• Future Direction: The study highlights the need for structural validation layers when using LLMs for model generation and sets a benchmark for future AI-driven testing in safety-critical systems.

In conclusion, the paper establishes that with the right prompting strategies (Few-shot) and configuration (Low-Medium temperature), LLMs can generate high-quality, diverse, and executable mutants for Simulink-Stateflow models significantly faster and more effectively than conventional methods.