💻 computer science

A Comprehensive Study on Large Language Models for Mutation Testing

This paper presents a comprehensive empirical study demonstrating that while Large Language Models significantly outperform rule-based approaches in generating diverse, behaviorally accurate mutants with 111.29% higher fault detection rates, they simultaneously incur higher costs in terms of non-compilability, duplication, and equivalent mutant rates.

Original authors: Bo Wang, Mingda Chen, Ming Deng, Youfang Lin, Mark Harman, Mike Papadakis, Jie M. Zhang

Published 2026-01-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bo Wang, Mingda Chen, Ming Deng, Youfang Lin, Mark Harman, Mike Papadakis, Jie M. Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a software engineer trying to find bugs in a complex piece of code. To test how good your safety net (your test suite) is, you decide to play a game of "Find the Fake." You intentionally introduce small, subtle mistakes into the code to see if your tests catch them. In the software world, these intentional mistakes are called mutants.

For years, engineers used a "rulebook" to create these mistakes. It was like a robot following a strict checklist: "Change every plus sign to a minus sign," or "Swap this number for zero." It was fast and reliable, but the mistakes it made were often obvious and didn't look like the messy, human errors that actually happen in real life.

Recently, a new player has entered the game: Large Language Models (LLMs). Think of these as super-smart AI assistants that have read almost every piece of code ever written. They don't just follow a rulebook; they understand context, logic, and style. They can look at a piece of code and say, "Ah, if I change this variable here, it looks exactly like a mistake a tired developer might make."

This paper is a massive "taste test" to see who is better at creating these fake mistakes: the old-school rulebook robots or the new AI assistants.

The Big Experiment

The researchers gathered 851 real-world bugs from popular Java software projects. These were actual mistakes that real developers had made and fixed. They then asked two groups to try to recreate these bugs:

The Traditionalists: Old-school tools (like PIT and Major) that follow strict rules.
The AI Team: Various Large Language Models (like GPT-4o and DeepSeek) using different "prompts" (instructions). One of these prompts was a new design created by the authors, called LLMut.

They generated over 700,000 fake mistakes (mutants) to see how they compared to the real ones.

The Results: The AI Wins on "Realism"

The findings were clear, but with a catch.

1. The AI is Better at Mimicking Real Bugs
Imagine you are trying to trick a security guard.

The Traditionalists put up a cardboard cutout of a burglar. It's a fake, but it's obviously fake. The guard (the test suite) spots it immediately, but it doesn't tell you much about how a real burglar would sneak in.
The AI creates a burglar that looks, walks, and talks exactly like a real one. It even wears the right clothes.

The paper found that the AI-generated mutants were 1.75 times better at tricking the tests than the traditional tools. Specifically:

Traditional tools caught about 44% of the real bugs.
AI tools caught about 76% of the real bugs.

The AI mutants were "behaviorally closer" to real bugs. They didn't just break the code in obvious ways; they broke it in the same subtle, confusing ways that humans do. This is huge because it means the AI is helping engineers find the real weak spots in their software.

2. The AI is More Creative
The traditional tools mostly made tiny, simple changes (like changing one number). The AI, however, was like a creative writer. It made complex changes, rearranging logic and structure in ways the rulebooks never thought of. It introduced a much wider variety of "mistakes," covering more ground.

The Catch: The AI is Messy

While the AI was better at creating realistic bugs, it was also much messier.

Compilation Errors: The traditional tools were like a perfect printer; almost every piece of paper came out readable. The AI, however, was like a student writing an essay: it often made typos or forgot to close a parenthesis, resulting in code that wouldn't even run. About 32% of the AI's attempts were "non-compilable" (broken code), compared to almost 0% for the traditional tools.
Duplicates: The AI sometimes got bored or confused and generated the exact same mistake twice, or even generated a mistake that looked like the original code (a "duplicate"). The traditional tools rarely did this.
Cost: The AI took longer and used more "tokens" (the currency of AI computing) to generate a single mutant compared to the lightning-fast traditional tools.

The "Surviving" Mutants

A key part of mutation testing is looking at the mutants that survive (the tests didn't catch them).

Traditional tools often created mutants that were so subtle they slipped through, but they were often in code areas that were already being tested.
AI tools tended to create mutants in untested areas of the code. This is a goldmine for engineers. It's like the AI pointing a flashlight at the dark corners of the room and saying, "Hey, nobody is checking this part yet. You should probably write a test for it."

The Verdict

The paper concludes that LLMs are a powerful new tool for software testing. They create mutants that are far more realistic and diverse than anything we've had before, helping engineers find deeper flaws in their software.

However, they aren't ready to replace the old tools entirely yet. They are too prone to making "typos" (compilation errors) and are slower. The ideal future, according to the paper, is a hybrid approach: use the AI to generate the realistic, creative bugs, but use the traditional tools to ensure the code is clean and valid.

In short: The AI is the creative genius who comes up with brilliant, realistic ideas but needs an editor to fix the grammar. The traditional tools are the reliable editors who never make mistakes but lack creativity. Together, they make the perfect team.

Technical Summary: A Comprehensive Study on Large Language Models for Mutation Testing

Problem Statement

Mutation testing is a critical technique for evaluating test suite quality, yet its industrial adoption has historically been limited by the high cost of generating effective mutants. While Large Language Models (LLMs) have recently been deployed in industry (e.g., by Meta and Google) to generate mutants that better resemble real faults, there has been no large-scale, comprehensive empirical study validating their performance against traditional methods. The field lacks a reliable baseline for the state-of-the-art effectiveness and efficiency of LLM-based mutation testing. Furthermore, the generation of mutants faces three persistent challenges: (1) generating mutants that effectively mimic real faults (coupling), (2) ensuring mutants are syntactically and semantically valid (compilability), and (3) ensuring mutants are diverse and killable while avoiding equivalent or duplicate mutants.

Methodology

The authors conducted a comprehensive empirical study comparing traditional rule-based and learning-based approaches against various LLM-based strategies.

Approaches Evaluated:
- Traditional: Four widely studied methods: PIT (rule-based), Major (rule-based), LEAM (learning-based sequence-to-sequence), and $\mu$ BERT (pre-trained BERT-based).
- LLM-based: Two existing state-of-the-art approaches (BugFarm and LLMorpheus) and a newly designed prompt template called LLMut.
- Models: The study utilized a diverse set of 12 LLMs, including closed-source models (GPT-3.5, GPT-4o, GPT-4o-Mini) and open-source models (DeepSeek-V3-671b, DeepSeek-Coder-V2-236b, StarChat-16b, CodeLlama-Instruct-13b).
Datasets: Experiments were conducted on 851 real-world Java bugs from two benchmarks:
- Defects4J 2.0: 605 bugs from 12 open-source projects.
- ConDefects: 246 bugs from recent programming contests (AtCoder) to mitigate data leakage concerns regarding model training data.
Metrics: The study evaluated mutants across three dimensions:
- Effectiveness: Real bug detection rate, Average Ochiai coefficient (semantic similarity), Coupling Rate, and AST node diversity.
- Validity: Compilability rate, Duplicate Mutant Rate, and Equivalent Mutant Rate (manually verified via sampling).
- Efficiency: Generation time and token consumption.
Experimental Design: The study involved generating over 701,400 mutants. It included sensitivity analyses on prompt engineering (context length, few-shot examples), high-order mutant generation, and root cause analysis of non-compilable mutants.

Key Contributions

Comprehensive Evaluation: The first large-scale empirical study comparing LLM-based mutant generation against traditional rule-based and learning-based approaches across multiple models and datasets.
Novel Prompt Design (LLMut): Introduction of a new prompt template incorporating few-shot examples from real bugs and structured JSON output, which was evaluated across multiple models.
High-Quality Dataset: Construction of a dataset of Java mutants annotated with detailed quality and behavior metrics, made available for future research.
Error Analysis: A detailed categorization of compilation errors in LLM-generated mutants, identifying specific code contexts (e.g., method invocations) prone to failure.

Key Results

Effectiveness

Superior Bug Detection: LLM-based approaches significantly outperformed traditional methods in detecting real bugs. LLM-based approaches achieved a 76.47% real bug detection rate compared to 44.15% for rule-based techniques (an absolute gain of 32.99 percentage points).
Best Performer: LLMut (DS-671b) achieved the highest detection rate at 91.1%, compared to 66.9% for Major and 40.1% for PIT.
Semantic Similarity: LLMs generated mutants with higher behavioral similarity to real bugs. LLMorpheus (DS-671b) achieved the highest coupling rate (52.0%), and LLMut variants showed significantly higher Average Ochiai coefficients (up to 59.8%) compared to traditional methods (max 35.5% for Major).
Diversity: LLMs introduced significantly more diverse syntactic changes. LLMut (DS-671b) and LLMut (DS-236b) introduced 49 new AST node types, whereas the traditional rule-based approach Major introduced only 2. LLMs also performed more complex code transformations, with a lower proportion of mutants having an AST edit distance of 1 compared to traditional approaches.

Validity

Compilability Trade-off: While LLMs were more effective, they suffered from lower compilability. Rule-based approaches (PIT and Major) achieved near-perfect compilability (100% and 97.6%, respectively). In contrast, LLM-based approaches ranged from 16.4% (StarChat) to 77.6% (LLMut DS-671b).
Redundancy: LLMs generated significantly more duplicate mutants (average 9.38% for LLMs vs. 0% for rule-based) and slightly higher equivalent mutant rates (average 4.74% for LLMs vs. 1.10% for rule-based).
Error Types: The most prevalent compilation errors in LLM-generated mutants were "Usage of Unknown Methods" (27.3%) and "Code Structural Destruction" (22.9%). These errors were most frequently associated with MemberReference and MethodInvocation contexts.

Efficiency

Time and Cost: Rule-based approaches were orders of magnitude faster (e.g., PIT: 0.02s/mutant) compared to LLMs (ranging from 1.31s to 9.06s/mutant). Token consumption varied significantly based on prompt engineering; the proposed LLMut prompt was more token-efficient than BugFarm and LLMorpheus due to its structured output requirements.

Surviving Mutants

Analysis of surviving mutants revealed that LLM-based approaches are more likely to generate mutants in uncovered code regions (Not Covered), whereas traditional approaches generated more mutants that were covered but failed to propagate state changes (Not Injected) or were undetected. This suggests LLMs are particularly valuable for guiding test suite augmentation to increase coverage.

Significance and Claims

The paper claims that LLMs represent a significant advancement in mutation testing effectiveness, generating mutants that are behaviorally closer to real bugs and more diverse than traditional methods. This allows practitioners to have greater confidence in deploying LLM-based mutation for improving test suites.

However, the authors modestly note that these effectiveness gains come at a cost: higher rates of non-compilable, duplicate, and equivalent mutants, as well as increased computational time. The study provides a baseline for the state-of-the-art, enabling researchers to develop techniques that further improve effectiveness while mitigating costs (e.g., through better prompt engineering or hybrid approaches). The authors emphasize that while LLMs excel at mimicking real bugs, future work must address syntactic validity and the specific error patterns (particularly in method invocations) to make these approaches robust for industrial-scale deployment.