Technical Summary: Large Language Models for Multi-Lingual Equivalent Mutant Detection

1. Problem Statement

Mutation testing is a robust technique for evaluating test suite effectiveness by introducing artificial faults (mutants) into source code. However, its practical adoption is hindered by the equivalent mutant problem: mutants that are syntactically different but semantically identical to the original program. These "equivalent mutants" cannot be killed by any test case, leading to inflated computational costs and biased mutation scores (preventing the metric from reaching 100% even for perfect test suites).

While numerous Equivalent Mutant Detection (EMD) methods exist, they face distinct limitations:

• Pure-code analysis methods (e.g., constraint-based testing, compiler optimization) rely on specific compiler infrastructures or manual heuristics.
• Traditional Machine Learning (ML) and Tree-based Neural Network (NN) approaches struggle with scarce labeled training data and limited generalization to unseen mutant patterns, often failing to capture deep program semantics.

Although Large Language Models (LLMs) have demonstrated superior semantic understanding in various software engineering tasks, their potential for EMD, particularly in a multi-lingual context, remains largely unexplored. Previous work by the authors established LLM superiority in Java but left critical gaps regarding other languages (e.g., C), the breadth of model architectures, and cross-lingual generalization.

2. Methodology

This study presents a comprehensive empirical evaluation of LLMs for EMD across Java and C programming languages.

2.1 Dataset

The authors utilized MutantBench, a widely-used benchmark containing 4,400 mutant pairs. The study focused on:

• Java: 3,302 method-level mutant pairs from 19 programs.
• C: 1,088 method-level mutant pairs from 18 programs.
• Data was pre-processed to the method level (aligning with unit testing workflows) and stripped of natural language comments. Stratified sampling created balanced training and test sets.

2.2 Models and Strategies

The study evaluated 13 state-of-the-art LLMs, categorized into encoder-based (e.g., CodeBERT, UniXCoder, CodeT5+) and decoder-only (e.g., Code Llama, Llama 3, Qwen2.5-Coder, DeepSeek-Coder, GPT-4) architectures.

Four distinct LLM adaptation strategies were investigated:

1. Pre-trained Code Embedding: Using the encoder component of LLMs to generate embeddings, followed by a fixed encoder and a trainable Multi-Layer Perceptron (MLP) classifier.
2. Fine-tuned Code Embedding: Similar to above, but with joint optimization of both encoder and classifier parameters.
3. Prompting Strategies (Zero-shot & Few-shot): Using structured natural language instructions to ask the LLM to determine semantic equivalence without parameter updates.
4. Fine-tuning with Instruction: Training decoder-only models on datasets containing structured instructions (mutant pair + instruction + label) to align with user intent.

2.3 Baselines and Metrics

The LLM approaches were benchmarked against 10 established EMD techniques:

• Compiler-based: Trivial Compiler Equivalence (TCE) using various compilers (javac, Soot, gcc, clang).
• ML-based: Seven classifiers (KNN, DT, RF, SVM, LDA, LR, GNB) using code features.
• Tree-based NN: Abstract Syntax Tree Neural Network (ASTNN).

Performance was measured using Precision, Recall, and F1-score (macro-averaged to handle class imbalance). Efficiency was assessed via inference time and training time.

3. Key Contributions

This paper extends the authors' prior work (ISSTA 2024) by:

1. Extending the Language Scope: Moving from a single-language (Java) study to a multi-lingual evaluation including C.
2. Expanding Model Coverage: Incorporating the latest LLMs (DeepSeek-Coder, Llama 3, Qwen2.5-Coder) to ensure findings are not model-specific.
3. Cross-Lingual Generalization: Investigating whether models trained on mixed-language datasets can generalize to single-language inference tasks.
4. Comprehensive Analysis: Providing a deep dive into strategy orthogonality, efficiency trade-offs, and qualitative failure modes.

4. Experimental Results

4.1 Effectiveness (RQ1 & RQ2)

• Superiority over Baselines: LLM-based approaches achieved significantly higher F1-scores than all 10 traditional baselines.
  • In Java, LLMs improved F1-scores by 75.18% over Compiler-based, 19.14% over ML-based, and 12.75% over Tree-based NN techniques.
  • In C, improvements were 557.79% (Compiler), 7.21% (ML), and 48.83% (Tree-based NN).
• Best Strategy: Fine-tuned code embedding consistently outperformed prompting strategies.
  • Java: Fine-tuned UniXCoder achieved the highest F1-score (86.58%), improving by 1.16%–84.10% over other combinations.
  • C: Fine-tuned CodeT5+ yielded the best results, with F1-score improvements of 0.08%–471.96% over alternatives.
• Prompting Limitations: Strategies relying solely on prompting (zero-shot/few-shot) yielded significantly lower metrics compared to embedding-based approaches.

4.2 Orthogonality (RQ3)

• LLM-based techniques and fine-tuned code embedding strategies demonstrated high orthogonality, identifying unique correct detections that other categories missed.
• They also produced fewer unique incorrect detections (false positives) compared to Compiler-based and Tree-based NN methods, particularly in the C environment where LLMs had zero unique incorrect detections.

4.3 Efficiency (RQ4)

• Inference Time: LLM-based techniques (e.g., UniXCoder at ~0.04s for Java) are significantly faster than Compiler-based techniques (e.g., TCE at ~2.35s for Java) but slightly slower than lightweight ML models (e.g., KNN at ~0.002s).
• Trade-off: The study concludes that LLMs offer a practical balance, accepting a minor increase in inference time for a substantial gain in detection accuracy.

4.4 Cross-Lingual Generalization (RQ5)

• Instruction Tuning: Fine-tuning with instructions on cross-lingual datasets (Java + C) improved performance for both languages, yielding 3.55%–3.70% higher F1-scores compared to single-language fine-tuning.
• Embedding Strategy: Conversely, fine-tuned code embedding strategies performed worse when trained on cross-lingual data, suggesting that for embedding-based approaches, language-specific tuning is preferable.

4.5 Qualitative Analysis of Failures

The authors identified specific failure modes where LLMs struggled compared to traditional methods:

1. Type-Coercion/Side Effects: Misinterpreting pre/post-increment operators in type-casting contexts.
2. Operator Coverage Mismatch: Failing to reason about the cardinality of value ranges covered by relational operators (e.g., >= vs ==).
3. Program Invariants: Missing constraints established by earlier statements in the method.
4. Algebraic Identity: Failing to recognize complex arithmetic equivalences.

• Observation: In these cases, Compiler-based techniques often succeeded where LLMs failed, highlighting the need for hybrid pipelines.

5. Significance and Claims

The paper claims that LLMs represent a viable and efficient solution for the longstanding challenge of equivalent mutant detection. Key takeaways include:

• State-of-the-Art Performance: LLMs, particularly when fine-tuned with code embeddings, outperform all existing traditional EMD methods in both Java and C.
• Practical Viability: Despite higher computational costs than simple ML models, LLMs provide a necessary trade-off, offering high accuracy with inference times compatible with real-time IDE warnings or automated pipelines.
• Cross-Lingual Potential: Fine-tuning with instructions enables models to generalize across structurally similar languages (Java/C), potentially reducing the maintenance overhead of language-specific models.
• Future Direction: The authors suggest that LLMs should not replace but rather complement traditional techniques. A hybrid pipeline—using compiler-based methods for initial filtering and LLMs for complex semantic analysis—is proposed as the most promising path forward.

The study establishes a new baseline for EMD research and provides open-source data and code to facilitate reproducibility and further advancement in mutation testing.