💻 computer science

A semantic mutation metric for metamorphic relation adequacy in scientific computing programs

This paper proposes the Semantic Mutation Score (SMS), a backward-compatible metric utilizing five domain-semantic operators to address the limitations of classical syntactic mutation testing in scientific computing, demonstrating through a large-scale study that while LLM-generated semantic mutants offer distinct coverage beyond traditional AST-based methods, they yield a medium rather than large effect size in adequacy assessment.

Original authors: Meng Li (School of Computing, University of South China, Hengyang, 421001, China, Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, 421001, Chi

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Meng Li (School of Computing, University of South China, Hengyang, 421001, China, Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, 421001, China, CNNC Key Laboratory on High Trusted Computing, Hengyang, 421001, China), Xiaohua Yang (School of Computing, University of South China, Hengyang, 421001, China, Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, 421001, China, CNNC Key Laboratory on High Trusted Computing, Hengyang, 421001, China), Jie Liu (School of Computing, University of South China, Hengyang, 421001, China, Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, 421001, China, CNNC Key Laboratory on High Trusted Computing, Hengyang, 421001, China), Shiyu Yan (School of Computing, University of South China, Hengyang, 421001, China, Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, 421001, China, CNNC Key Laboratory on High Trusted Computing, Hengyang, 421001, China)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Finding Hidden Bugs in Math Software

Imagine you are building a complex machine that solves difficult math problems (like predicting weather or simulating physics). You want to make sure it doesn't have any hidden bugs.

The problem is: How do you know if the answer is right?
In normal software, you check the answer against a "key" (like a math textbook). But in advanced scientific computing, there often isn't a textbook answer. The math is too complex. This is called the "Test Oracle Problem."

To solve this, researchers use Metamorphic Testing (MT). Instead of checking if the answer is "correct," they check if the answer follows the rules of the universe.

Analogy: If you double the amount of ingredients in a cake recipe, the cake should be twice as big. If you double the ingredients and the cake stays the same size, something is wrong, even if you don't know the exact size of the perfect cake.

The New Tool: "Semantic Mutation Score" (SMS)

The authors created a new way to test these rules. They call it Semantic Mutation Score (SMS).

Think of the software code as a house.

Old Way (Syntactic Mutation): Imagine a robot that randomly swaps bricks, changes the color of the paint, or moves a window one inch to the left. This is "syntax." It changes the look of the house but usually doesn't break the house's structure.
New Way (Semantic Mutation): Imagine a robot that changes the physics of the house. It removes a load-bearing wall, changes the foundation from concrete to jelly, or swaps a steel beam for a rubber band. This is "semantics." It breaks the logic of the house.

The authors built five special "bug-injecting" robots (operators) designed to break the specific logic of scientific math:

Conservation Erosion: Breaking the rule that "matter/energy can't be created or destroyed."
Operator Substitution: Swapping a math tool for a different one that looks similar but does something else (like using a hammer to screw in a bolt).
Hyperparameter: Changing a "knob" that controls how the math works (like turning a dial from "slow and steady" to "fast and sloppy").
Trajectory Flip: Making a path go backward or sideways when it should go forward.
Structural Injection: Adding a new part to the machine that shouldn't be there.

The Experiment: 12 Test Kitchens

The researchers tested their new robots on 12 small "test kitchens" (Programs Under Test). These kitchens represent different types of math:

Numeric: Solving equations.
Probabilistic: Rolling dice and calculating odds.
Surrogate: Using a simple model to guess a complex result.
Machine Learning: Teaching a computer to learn patterns.

They used Large Language Models (LLMs) (like the AI you are talking to) to generate these "buggy" versions of the code. They asked the AI: "Here is a math program. Please break its logic in a specific way."

The Results: What They Found

1. The AI is good at finding "Deep" bugs.
When they compared the AI-generated bugs to standard, old-school robot bugs (which just swap math symbols), they found a huge difference.

The Overlap: Only 5% of the AI bugs were the same as the old-school bugs.
The Gap: The AI found 54% of the bugs that the old robots could not even see. These were bugs involving changing the "knobs" (Hyperparameters), flipping the path (Trajectory), or changing the structure (Structural Injection). The old robots were blind to these because they only looked at the surface code, not the deep meaning.

2. The "Effect Size" was Medium, not Huge.
The researchers had a big hope: They thought that using different AIs (Claude, GPT, DeepSeek) would create a massive variety of bugs, making the test much stronger.

The Reality: Using different AIs didn't change the results much. Whether they used one AI or three, the "score" of how well the tests worked stayed roughly the same.
The Analogy: It's like asking three different chefs to break a cake. You might expect them to break it in totally different ways, but they all ended up smashing it in the same spot. The quality of the broken pieces was slightly better with more chefs, but the pattern of the break didn't change.

3. The "Zero" Problem.
In 75% of the test cases, the new system found zero bugs.

Why? It wasn't because the code was perfect. It was because the "rules" (Metamorphic Relations) they were testing against were too loose. The bugs existed, but the specific rules they were checking didn't catch them. It's like having a net with holes too big to catch small fish.

The Conclusion: A Better Ruler, But Not a Magic Wand

The paper concludes that Semantic Mutation Score (SMS) is a valid, backward-compatible tool.

Backward-Compatible: If you turn off the "smart" features and just look at simple code swaps, it works exactly like the old, trusted Mutation Score.
The Verdict: The new method successfully identifies a class of deep, logical bugs that standard tools miss. However, the researchers admit that simply using more AI models doesn't automatically make the test perfect. The real key is designing better "rules" (Metamorphic Relations) to catch the bugs the AI finds.

In short: They built a new, smarter bug-hunter that understands the meaning of math code, not just the letters. It finds bugs the old hunters miss, but it still needs better instructions (rules) to catch everything.

Technical Summary: A Semantic Mutation Metric for Metamorphic Relation Adequacy in Scientific Computing Programs

1. Problem Statement

Metamorphic Testing (MT) addresses the test-oracle problem in scientific computing by verifying invariants (Metamorphic Relations, or MRs) between related inputs rather than checking outputs against known correct values. However, assessing the adequacy of an MR set against domain-specific faults has lacked a metric tied to the classical Mutation Score (MS) lineage.

Classical MS relies on syntactic Abstract Syntax Tree (AST) mutations (e.g., operator replacement, constant modification). These tools fail to capture domain semantics critical to scientific computing, such as conservation laws, monotonicity, convergence orders, trajectory shapes, and fidelity ordering. Furthermore, while Large Language Models (LLMs) can generate semantically richer mutants, prior work has not isolated the contribution of LLM source diversity from the design of the MRs themselves.

2. Methodology

2.1 The Semantic Mutation Score (SMS)

The authors propose Semantic Mutation Score (SMS), a metric designed as a strict generalization of classical MS. SMS is built upon a three-layer framework:

Layer 1 (Definitional): A mutant is defined as "semantic" if it satisfies at least one of three necessary conditions:
1. Cross-boundary: It crosses a function-call or module-import boundary.
2. Domain-knowledge dependent: Its legality depends on mathematical, physical, or statistical knowledge (not just type preservation).
3. Algorithmic class change: It alters the algorithmic class of the implementation.
  Based on these, five meta-operator classes are defined:
- Conservation Erosion (CE): Breaking conservation laws.
- Operator Substitution (OS): Replacing operators with semantically distinct alternatives (e.g., det vs. sum(diag)).
- Hyperparameter (HP): Modifying parameters with specific domain semantics (e.g., noise levels, learning rates).
- Trajectory Flip (TF): Distorting numerical trajectories or integration orders.
- Structural Injection (SI): Injecting structural changes affecting fidelity or order.
Layer 2 (Operational): An equivalence judgment ( $E_1 \land E_2$ ) determines if a mutant is equivalent to the original.
- $E_1$ (AVP-coherent): The mutant passes the Automated Verification Pipeline (AVP) for all MRs.
- $E_2$ (Output-equivalent): The mutant produces outputs within a tolerance $\epsilon_{eq}$ on $K_{eq}=1000$ samples.
- A mutant is "killed" if it fails the AVP for at least one MR.
Layer 3 (Applied): Empirical traceability comparing LLM-generated mutants against syntactic mutants (using the cosmic-ray tool) at the AST-normalized level.

2.2 Experimental Design

Subjects: 12 Programs Under Test (PUTs) representing four classes of single-output scientific kernels ( $float \to float$ ): Numeric (A), Probabilistic (B), Surrogate (C), and Machine Learning (D). Each PUT is under 2 KB.
Design: A $12 \text{ PUTs} \times 5 \text{ Operators} \times 5 \text{ Metamorphic Patterns}$ matrix, resulting in 60 experimental cells.
LLM Protocol:
- v3 (Same-source): Mutants generated by a single LLM (Claude Opus).
- v4 (Cross-source): Mutants generated by three LLMs (Claude, GPT-5.4, DeepSeek) under an identical prompt to isolate the effect of LLM source diversity.
Attribution (LRCA): A three-layer classifier (Likely Root Cause Attribution) categorizes killed mutants into:
- C1: True semantic failure.
- C2–C5: Numerical tolerance, Out-of-Distribution (OOD), statistical violations, or mutator artifacts.
- Metrics include $C1\_share$ (proportion of true semantic failures) and $suspect\_share$ .

2.3 Backward Compatibility

The paper proves that under a degenerate limit ( $L$ ) where operators become rule-based syntactic, equivalence becomes bitwise, and programs are deterministic imperative, SMS reduces almost everywhere to the classical Mutation Score (Theorem 9.1). This ensures structural consistency with existing mutation testing literature.

3. Key Results

3.1 Semantic Reachability vs. Syntactic Tools

AST Overlap: There is only a 5.14% AST overlap between LLM-generated mutants and default cosmic-ray syntactic mutants across the 12 PUTs.
Unreachability: Three operator classes—Hyperparameter (HP), Structural Injection (SI), and Trajectory Flip (TF)—are 0% reachable by default first-order syntactic tools (0/72, 0/33, and 0/54 overlaps, respectively). These constitute 54.5% of the LLM mutant pool.
Conclusion: Default syntactic tools (even with higher-order compositions, though not empirically tested) cannot systematically generate these domain-specific faults.

3.2 Effect Size and LLM Source Diversity

Cliff's Delta ( $\delta$ ): The study compares "aligned" slices (operator matches MR pattern) vs. "cross" slices (mismatched).
- v3 (Same-source): $\delta = 0.323$ (95% CI [0.017, 0.622]).
- v4 (Cross-source): $\delta = 0.314$ (95% CI [0.014, 0.622]).
Hypothesis H2: The pre-registered large-effect threshold ( $\delta \ge 0.474$ ) was not met. The observed effect lies in the medium-effect range.
Source Diversity: Switching from a single-source to a cross-source pool under an identical prompt shifted $\delta$ by only -0.009 (CI covers zero). This indicates that LLM identity is not the primary lever for the aligned-vs-cross effect size in this design.

3.3 Mutant Quality and LRCA

Quality Improvement: Cross-source pooling increased the mean $C1\_share$ (true semantic failures) from 16.4% to 20.9% (+27%).
Hypothesis H4: The pre-registered threshold of mean $suspect\_share \le 0.20$ (i.e., $C1\_share \ge 0.80$ ) was not met. The observed mean $C1\_share$ was 0.209 (best calibrated), meaning 79.1% of kills were attributed to non-semantic causes (tolerance, OOD, etc.).
Decoupling: A sharp decoupling was observed between Semantic Feasibility ( $R_{sem}$ ) and Kill Rate ( $R_{kill}$ ). Many mutants were syntactically valid and semantically intended ( $R_{sem} \approx 1.0$ ) but failed to be killed by the primary MRs ( $R_{kill} = 0$ ), suggesting that MR design adequacy, not just mutant generation, limits SMS.

3.4 Statistical Power

Power Analysis: The study acknowledges low statistical power due to the sample size ( $N=12$ PUTs, 60 cells). The stipulated power to detect a large effect ( $\delta \ge 0.474$ ) was only 49.1%. The authors frame the "not met" verdict as a factual statement about the point estimate failing the threshold, not a definitive proof that the effect is small.

4. Significance and Claims

The paper claims the following contributions:

Formal Metric: It introduces SMS, the first domain-specific semantic mutation score for MR adequacy in scientific computing kernels, with a proven backward-compatible reduction to classical MS.
Operator Framework: It defines and implements five semantic mutation operators (CE, OS, HP, TF, SI) that target faults unreachable by default syntactic tools.
Empirical Evidence of Unreachability: It provides empirical evidence that default first-order syntactic tools cannot generate HP, SI, and TF mutants, validating the need for semantic operators.
LLM Source Diversity Findings: It demonstrates that under an identical prompt, LLM source diversity does not appreciably shift the effect size (Cliff's delta) of aligned-vs-cross MR adequacy, though it does improve the quality (C1 share) of the kill set.
Methodological Framework: It establishes a three-layer framework (Definitional, Operational, Applied) for evaluating semantic mutation, including a conservative equivalence judgment ( $E_1 \land E_2$ ) and an engineering attribution layer (LRCA).

Limitations and Modesty:
The authors explicitly state that the study is an underpowered exploratory contribution. The "not met" verdict for H2 is attributed to the point estimate failing the threshold, with the possibility that a larger sample size (P4) might reveal a different effect size. They do not claim that LLMs are the ultimate solution for test generation, nor do they claim SMS is ready for normative industrial certification (e.g., IEC 60880, DO-178C) due to air-gap incompatibilities and the need for further scaling. The work is positioned as a foundational step (P2) in a five-paper roadmap, with formal theorems and industrial transfer reserved for companion papers (P4, P5).