Using Mutation-Analysis to Examine an LLM's Ability to Summarize Code

Technical Summary: Using Mutation-Analysis to Examine an LLM's Ability to Summarize Code

Problem Statement
As developers increasingly rely on Large Language Models (LLMs) for code summarization to aid in documentation, testing, and review, a critical concern arises regarding the accuracy of these summaries. LLMs often generate confident descriptions based on the intended logic or familiar algorithmic patterns of code, rather than the code's actual behavior. This discrepancy is particularly dangerous when subtle changes, such as edge cases or minor logic modifications, alter what a program does without changing its superficial appearance. Existing evaluation methods, which rely on n-gram overlap metrics (BLEU, ROUGE) or LLM-based evaluators, often fail to correlate with human judgment regarding semantic correctness. Furthermore, there is a lack of systematic frameworks to determine if a generated summary truly tracks the underlying behavior of the code or merely reflects surface-level textual patterns.

Methodology: Mutation-Based Evaluation
The authors propose a mutation-based evaluation methodology to directly test whether an LLM's summary reflects the specific logic of the provided code. The core premise is that if a summary accurately describes code behavior, it must change when the code's behavior is altered. The experimental process involves three main stages:

1. Generate Mutants: Targeted mutations are injected into original code samples. These mutations are categorized into three types:

   • Statement mutations: Removing, duplicating, or rearranging statements to alter execution flow.
   • Value mutations: Modifying parameters or constant values.
   • Decision mutations: Changing arithmetic or logic operators to affect decision-making logic.
     Mutations are placed in specific locations (beginning, middle, end) to test attention mechanisms.

2. Generate Summaries: The LLM is prompted with a fixed instruction ("Explain the following code snippet in plain English") to generate summaries for both the original code and the mutated variants. Temperature is set to 0 to ensure deterministic output.

3. Compare Summaries: Human evaluators compare the summary of the mutated code against the original. A result is classified as Positive if the summary explicitly reflects the behavioral change (e.g., noting a different condition or return value). It is classified as Negative if the summary remains identical to the original description or describes the expected algorithmic intent rather than the actual mutated behavior.

Key Contributions
The primary contribution of this work is the establishment of mutation analysis as a systematic evaluation framework for code summarization. The authors validate this methodology through three complementary experiments:

1. Controlled Synthetic Dataset: 12 Python programs spanning four complexity categories (Single Function, Single Class, Multiple Classes, Multi-threaded) with 324 total mutations.
2. Human-Written Code Validation: Evaluation on 50 programs from the Less Basic Python Problems (LBPP) dataset to ensure findings generalize beyond model-generated code.
3. Cross-Model Comparison: A direct comparison between GPT-4 and GPT-5.2 using the same LBPP subset to measure generational improvements.

Key Results

• Impact of Complexity: For GPT-4, summary accuracy decreases sharply as program complexity increases. Accuracy drops from 76.5% for single-function programs to 17.3% for multi-threaded systems. Program size also correlates negatively with accuracy (median 60 LOC for positive cases vs. 116 LOC for negative cases).
• Mutation Type and Location: Mutation type and location generally exhibit weaker effects on accuracy compared to complexity. However, value mutations are often detected more reliably than statement or decision mutations, except in multi-threaded contexts where value mutations degrade significantly.
• Failure Modes: The analysis identifies two primary failure modes:
  1. Abstraction: The summary is too high-level, omitting the mutated logic entirely.
  2. Intent Hallucination: The model describes the expected behavior of a known algorithm (e.g., Merge Sort, Kruskal's) rather than the actual mutated code, ignoring deviations from canonical patterns.
• Human-Written Code: The failure patterns observed in synthetic data persist in human-written code (LBPP), with GPT-4 achieving a 49.3% mutation detection rate. This confirms that the limitations are not artifacts of model-generated code.
• Model Generations: GPT-5.2 demonstrates a substantial improvement over GPT-4, achieving an 85.3% detection rate on the LBPP dataset (a 36.0 percentage point increase). GPT-5.2 also shows a new capability to identify mutations as "bugs" or "errors" in nearly half of the detected cases. However, it still struggles with complex control flow changes and fails to distinguish implementation details from standard patterns in roughly 15% of cases.

Significance and Claims
The paper claims that mutation analysis provides a practical, repeatable stress test for assessing whether LLM-generated summaries reflect program behavior rather than superficial patterns. The authors position this methodology as a necessary precursor to improvement, offering a way to:

• Directly compare models, prompts, and datasets under controlled conditions.
• Verify whether improvements in summarization (via fine-tuning, prompting, or new models) are genuine rather than stylistic.
• Provide IDE developers and researchers with measurable metrics to determine if switching models or modifying tools yields tangible benefits for specific use cases.

The authors remain modest, noting that while GPT-5.2 shows significant progress, an 85.3% success rate on simple single-function code still leaves a non-trivial error rate that could be problematic for high-stakes systems. The work does not claim to solve the problem of code understanding but rather provides a rigorous method to measure the gap between an LLM's summary and the code's actual logic.