Multi-Agent Specification-based Metamorphic Testing of FMU-Based Simulations

Technical Summary: Multi-Agent Specification-based Metamorphic Testing of FMU-Based Simulations

Problem Statement
In modern industrial model-based design, the Functional Mock-up Interface (FMI) standard facilitates the exchange of simulation models as Functional Mock-up Units (FMUs) across diverse tools and partners. While this enables simulation-based verification, validating these models presents a significant challenge: the "test oracle problem." FMUs often encapsulate binaries without source code, and their internal logic is opaque, preventing traditional testing that relies on internal state inspection. Furthermore, explicit expected outputs are rarely defined, as testing is typically performed manually by domain experts.

Metamorphic Testing (MT) offers a solution by verifying systems lacking explicit oracles through Metamorphic Relations (MRs)—properties that define how outputs should change in response to specific input transformations. However, deriving effective MRs from functional and interface specifications remains a manual, error-prone process. Existing approaches lack a comprehensive, automated workflow that starts from specifications, generates MRs, and executes tests specifically for FMU-constrained environments. Additionally, while Large Language Models (LLMs) show promise in generating MRs, they often require heavy human validation and struggle with the specific constraints of FMU interfaces (e.g., limited exposed variables and continuous time-series data).

Methodology: AgenticMeta
The authors propose AgenticMeta, an LLM-powered multi-agent workflow designed to automate specification-based metamorphic testing for FMU-based simulations. The system orchestrates a "hub-and-spoke" architecture where a Coordinator Node manages a shared graph state and directs specialized agents through four iterative phases:

1. Extraction Phase:
   • Extractor Agent: Converts functional specifications (PDF) into structured Markdown and extracts system properties. It parses the modelDescription.xml from the FMU to identify inputs, outputs, parameters, and constraints. The output is a structured JSON containing test conditions and variable definitions.
2. MR Generation and Refinement Phase:
   • MR Generator Agent: Uses the extracted knowledge to propose candidate MRs. These are formatted in Given–When–Then (GWT) patterns:
     • Given: Initial conditions for seed inputs.
     • When: Metamorphic transformations (e.g., "increase," "ramp") applied to inputs.
     • Then: Expected output relations (e.g., "Eventually Increases," "Proportional to").
   • MR Refiner Agent: Validates the generated MRs against the original specifications and extraction data. It checks for factual correctness, causal validity, and testability. MRs that fail validation are either refined with feedback or dropped. This separation of generation and refinement aims to improve stability and accuracy.
3. Test Generation Phase:
   • Test Generator Agent: Converts the refined GWT patterns into concrete time-series input signals. It applies sampling algorithms to generate seed and follow-up inputs, ensuring transformations occur within valid simulation windows (e.g., near the start of the simulation).
   • Test Validator Agent: Performs boundary checks, data-type validation, and ensures the test cases adhere to simulation-specific constraints before execution.
4. Execution and Evaluation Phase:
   • Deterministic tools execute the generated time-series inputs against the FMU using the FMPy library.
   • The system evaluates output consistency against the MRs.
   • Mutation Analysis: To assess test quality without access to internal code, the system applies mutation operators (Mirror, Crossover, Polynomial) to the recorded output time series of passed tests. The mutation score is calculated based on the ability of the MRs to detect these injected faults.

Key Contributions

• Automated Multi-Agent Workflow: A novel pipeline that integrates requirement extraction, MR generation/refinement, and test execution specifically for FMU-based dynamic simulations.
• GWT-Structured MRs: The approach specifies MRs using Given–When–Then patterns, which facilitates LLM processing and ensures structured input transformations and output relations.
• End-to-End Feasibility: The system automates the entire process from specification to mutation analysis, addressing the gap in existing literature where LLMs are often used only for candidate generation without full workflow integration.
• Stability Mechanisms: By separating MR generation from refinement and using a coordinator to manage state, the approach mitigates the instability and non-determinism often associated with LLM-based pipelines.

Experimental Results
The approach was evaluated on a Lubricating Oil Cooling (LOC) system FMU across 10 independent sessions.

• Execution: The workflow successfully completed all phases (extraction, generation, execution, analysis) in all sessions.
• Coverage: The system achieved an average requirement coverage of 60.47%, generating an average of 15.8 MRs and 55.7 test cases per run.
• Test Quality: The average test pass rate was 74.10%. The authors note that failures were often due to temporal settling issues or tolerance ranges rather than fundamental logic errors.
• Mutation Score: The average mutation score was 0.56, indicating that the generated MRs successfully detected 56% of the injected output mutations.
• Runtime: The average time per test case was approximately 2.72 seconds for generation and 17.12 seconds for execution. The MR generation and refinement phase was identified as the primary bottleneck, averaging 43.95 seconds per accepted MR.

Significance and Limitations
The paper claims that AgenticMeta demonstrates the feasibility of automating metamorphic testing for FMU-based simulations, significantly reducing manual effort in test oracle derivation. The results suggest that the workflow can systematically generate meaningful MRs and executable tests, supporting the verification of dynamic simulation models.

However, the authors modestly acknowledge several limitations:

• Coverage Constraints: Not all requirements could be converted into valid MRs, particularly those describing generic behaviors or outputs not directly controllable by exposed FMU inputs.
• Dependency on Specifications: The quality of the output is directly tied to the clarity and completeness of the input functional specifications and the modelDescription.xml.
• Mutation Analysis Scope: The mutation score reflects sensitivity to output deviations rather than a full fault-detection capability against implementation defects, as mutations were applied to recorded outputs rather than the FMU binary.
• Generalizability: The evaluation was conducted on a single case study (LOC system), and findings may not generalize to models with vastly different control logic or interface structures.

The paper concludes that while the approach is a significant step toward automated verification, future work will focus on making the Coordinator an adaptive LLM agent and refining mutation operators to better align with MR semantics.