🤖 machine learning

Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)

This large-scale replication study reveals that while code coverage and mutation scores are unreliable indicators of real-bug detection for LLM-generated tests in scenarios where the code-under-test may already be buggy, they remain meaningful signals in regression-style settings, challenging prior conclusions about the dominance of test suite size as a confounder.

Original authors: Junda Zhao, Shurui Zhou, Eldan Cohen

Published 2026-07-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junda Zhao, Shurui Zhou, Eldan Cohen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a crime, but instead of looking for fingerprints or footprints, you are looking for "bugs"—hidden mistakes in computer code that make programs crash or act weirdly. For decades, software engineers have relied on two main clues to see if their tests are any good: Code Coverage and Mutation Score. Think of Code Coverage like a flashlight; it tells you how much of the dark room (the code) you've shone your light on. If you've lit up 100% of the room, you feel confident you haven't missed anything. The Mutation Score is a bit more like a "stress test" or a "trap." Imagine someone secretly replaces a few bricks in a wall with weak, fake ones (these are the "mutations"). If your test knocks the wall down, it means your test is sharp enough to spot the weak spots.

For a long time, the big question in the world of software testing was: "Does shining a bright flashlight or knocking down a fake wall actually mean you'll find the real criminal?" Some older studies suggested that once you account for how many tests you ran, these clues stopped being very helpful. They argued that just because you covered more ground or killed more fake bugs, it didn't necessarily mean you were better at finding real, hidden errors. Now, a new player has entered the game: Large Language Models (LLMs). These are super-smart AI chatbots that can write code and, more recently, write tests for other code. But because these AI bots work differently than human detectives or old-school automated tools, we don't know if the old clues (flashlights and fake walls) still work for them. Do these AI-generated tests actually find real bugs, or are they just good at lighting up the room and knocking over fake bricks?

This paper is a massive detective story where the authors, Junda Zhao, Shurui Zhou, and Eldan Cohen, decided to put the old clues to the test again, but this time with AI-generated tests. They took 11 of the most advanced AI models available and asked them to write over 100,000 tests for real-world software projects. They then checked if the "flashlight" (coverage) and the "fake wall" (mutation score) actually predicted whether the AI found the real bugs.

Here is the twist: The results were surprisingly different from what everyone expected. The authors found that the old rules don't quite apply to AI. When the code the AI was testing was known to be clean (like a crime scene with no crime yet, just waiting for a future mistake), the old clues worked surprisingly well. If an AI model generated tests that covered more of the code or killed more fake mutations, it was indeed better at finding real bugs later on. In this specific scenario, the flashlight and the stress test were reliable guides for comparing which AI was the better detective.

However, the story changes completely when the code being tested was already broken. In the real world, we often ask AI to find bugs in code that is already messy. The authors discovered that in this messy scenario, the flashlight and the fake walls stopped working. Even if an AI lit up 100% of the code or knocked down every fake brick, it didn't mean the AI would actually find the real bug hiding in the mess. In fact, the AI sometimes got tricked by the broken code and wrote tests that celebrated the mistake instead of catching it. So, if the code is already buggy, the old metrics become unreliable; they can't tell you if the AI is actually good at finding errors.

Another major surprise was about the size of the test team. Previous studies had argued that the number of tests was the biggest trickster, making it look like bigger teams were better just because they had more people. But this paper suggests that for AI, the number of tests isn't the main driver. Whether an AI wrote 3 tests or 10, the relationship between how well it covered the code and how well it found bugs stayed roughly the same. The size of the team wasn't the magic ingredient; it was more about how the AI was thinking about the code.

In short, the paper suggests that we can't just blindly trust the old metrics when using AI. If you are testing clean code to catch future mistakes, coverage and mutation scores are still useful tools to compare different AI models. But if you are trying to find bugs in code that is already broken, those numbers might be lying to you. The authors conclude that we need to be much more careful about what we are testing and why, rather than just counting how many tests an AI wrote or how much code it touched. They didn't solve the mystery of how to make AI perfect at finding bugs, but they did clear up a lot of confusion about how to measure if an AI is doing a good job.

Technical Summary: Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)

Problem Statement
Recent advances in Large Language Models (LLMs) have spurred interest in automating test generation. However, the evaluation of LLM-generated test suites relies heavily on proxy metrics, primarily code coverage and mutation scores, rather than direct real-bug detection. Prior research on human-written tests (e.g., Inozemtseva et al., Papadakis et al.) suggests that once test suite size is controlled, the correlations between coverage, mutation, and real-bug detection effectiveness largely vanish. This raises concerns that evaluating LLM-generated tests via these proxies may be misleading. Furthermore, it is unclear whether these conclusions hold for LLM-generated tests, which are typically produced in a white-box setting where the code-under-test is provided as the primary prompt input. Crucially, existing LLM evaluation practices often assume the input code is bug-free (regression testing), whereas practical scenarios may involve buggy code where the goal is to expose existing faults.

Methodology
The authors conduct a large-scale conceptual replication of Inozemtseva et al. and Papadakis et al., adapted for the LLM era.

Benchmark: The study utilizes the Defects4J v3.0 benchmark, focusing on 318 buggy focal methods across 17 open-source Java projects. Each method has both a buggy and a fixed (bug-free) version.
Models: The evaluation covers 11 state-of-the-art LLMs from six major developers (including Google, Anthropic, OpenAI, xAI, DeepSeek, and Alibaba), encompassing both base and reasoning variants. This yields 13 distinct model settings.
Test Generation: A standardized, automated pipeline generates unit tests for each focal method using both its buggy and fixed versions. The workflow includes extracting surrounding code context (parameters, class fields, constructors) to construct prompts, followed by AST-based extraction of generated tests. This resulted in 8,268 test suites comprising over 101,000 individual test cases.
Experimental Design: To address confounding factors, the study employs random sampling strategies to construct test suites of varying sizes and fixed sizes ( $k \in \{3, 5, 10\}$ ).
Metrics & Analysis:
- Coverage: Measured using CodeCover (statement, branch, and modified condition coverage).
- Mutation: Measured using PIT (raw and normalized mutation scores).
- Effectiveness: Real-bug detection is defined as the ability of a test suite to pass on the fixed version but fail on the buggy version.
- Correlation Analysis: The authors analyze correlations across three granularities: (1) Combined (pooled across all models), (2) Intra-model (within a single model), and (3) Inter-model (comparing aggregate statistics across different models). They examine relationships between suite size, coverage, mutation, and bug detection under both unconstrained and size-controlled conditions.

Key Contributions

Large-Scale Replication: The study replicates foundational findings on test effectiveness using over 100,000 test cases generated by 11 SOTA LLMs, revealing that correlations for LLM-generated tests differ substantially from those reported for human-written tests.
Contextual Analysis of Buggy Code: The authors extend previous work by examining how coverage correlates with real-bug detection when the input code is buggy, a scenario often overlooked in LLM test generation evaluations.
Actionable Guidance: The paper provides specific guidance on interpreting coverage and mutation metrics in the LLM era, distinguishing between regression-style (bug-free input) and bug-exposure (buggy input) scenarios.
Artifact Availability: A complete replication package, including implementation, data, and analysis scripts, is provided to support future research.

Key Results

Suite Size as a Confounder: Contrary to prior studies on human-written tests, the authors find little evidence that test suite size is a dominant confounding factor for LLM-generated tests. Correlations between suite size and mutation/bug detection are weak to negligible.
Coverage and Mutation Correlations (Bug-Free Input):
- When the code-under-test is bug-free, coverage and mutation scores show weak correlations within a single model (intra-model) but moderate to strong correlations when comparing across different models (inter-model).
- This suggests that while coverage may not distinguish the quality of individual suites within a model, it is a meaningful signal for comparing the relative effectiveness of different LLMs.
- Different coverage types (statement, branch, modified condition) are not perfectly interchangeable; correlations between them are moderate to strong but not near-perfect, especially under accumulated aggregation.
Coverage and Mutation Correlations (Buggy Input):
- When tests are generated from buggy code, the correlation between coverage and real-bug detection becomes uniformly weak across all granularities.
- In this setting, coverage loses predictive power because LLMs may infer the buggy behavior as intended, generating tests that assert incorrect outcomes.
- Mutation analysis is deemed inapplicable in the buggy-code setting because it presupposes a passing test suite on the code-under-test.
Mutation and Bug Detection: Similar to coverage, mutation scores show weak correlations with bug detection in combined/intra-model views but exhibit moderate to strong correlations in the inter-model view (specifically raw mutation score), making it a useful metric for comparing models in bug-free settings.

Significance and Claims
The paper claims that the validity of coverage and mutation scores as proxies for test effectiveness is highly context-dependent.

For Regression Testing (Bug-Free Input): Coverage and mutation scores are informative and can serve as meaningful signals when comparing the effectiveness of different LLMs, even when test suite size is controlled.
For Bug Exposure (Buggy Input): These metrics are unreliable. In scenarios where the goal is to detect existing bugs in the code-under-test, coverage does not correlate with effectiveness, and mutation testing is not applicable.
Re-evaluating Prior Work: The authors argue that the discrepancy with prior studies (Inozemtseva et al., Papadakis et al.) stems from differences in test suite construction (LLM-generated suites reflect natural testing needs per method rather than random sampling from a project pool) and the unique nature of LLM prompting.
Recommendation: Future evaluations of LLM-based test generation should prioritize direct real-bug detection, especially when the input code may be buggy, rather than relying solely on proxy metrics. When using proxies, researchers should be cautious about interpreting results across different models versus within a single model and should consider the specific testing scenario (regression vs. bug exposure).

Technical Summary: Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)

More like this