🤖 AI

On the Impact of Code Comments for Automated Bug-Fixing: An Empirical Study

This empirical study demonstrates that retaining code comments during both training and inference significantly enhances Large Language Models' automated bug-fixing accuracy, challenging the common practice of removing comments and highlighting the value of implementation details in aiding model performance.

Original authors: Antonio Vitale, Emanuela Guglielmi, Simone Scalabrino, Rocco Oliveto

Published 2026-02-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Antonio Vitale, Emanuela Guglielmi, Simone Scalabrino, Rocco Oliveto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly literal-minded robot how to fix broken toys. This robot is a Large Language Model (LLM), and the "toys" are computer programs with bugs.

For a long time, researchers believed that to teach this robot, you should strip away all the "notes" or "stickers" (code comments) attached to the broken toys. They thought the robot should only look at the raw plastic and gears (the code itself) to figure out how to fix it.

The Big Idea of This Paper
The authors of this paper, Antonio Vitale and his team, asked a simple question: What if those notes are actually the most important part of the puzzle? They hypothesized that the notes written by the original human developers explain why the toy was built a certain way, which might be the key to fixing it.

To test this, they didn't just look at the raw code. They created a massive new training set where they used an AI to write high-quality "stickers" (comments) for every single broken toy. These stickers explained:

What the toy does.
Why it was built that way.
How the gears work inside.
How to use it properly.
What rules it must follow (like "don't drop this").

They then ran a series of experiments with two different types of robot teachers (CodeT5+ and DeepSeek-Coder) to see how well they could fix the bugs under four different scenarios:

No Notes: Training without notes, testing without notes.
Training Notes Only: Training with notes, testing without notes.
Testing Notes Only: Training without notes, testing with notes.
Notes Everywhere: Training with notes, testing with notes.

The Results: The Power of the "Stickers"

Here is what they found, using some simple analogies:

The "Notes Everywhere" Effect: When the robot was trained with the notes and then tested with the notes, it became a superhero. Its ability to fix bugs jumped by up to three times compared to when it had no notes at all. It was like giving the robot a manual and letting it read the manual while working.
The "Notes at the End" Effect: Even if the robot was trained on raw, note-less code, giving it the notes just when it was trying to fix a bug (at "inference time") still helped it significantly. It was like handing the robot the manual right when it got stuck.
The "No Harm" Rule: Interestingly, training the robot with notes didn't hurt its performance when the notes were missing later. It didn't get confused; it just performed slightly worse than its peak, but still better than robots that never saw notes at all.

Which Notes Matter Most?

The researchers also peeked inside the robot's "brain" to see which parts of the notes it was paying attention to. They found:

The "How" Notes are King: The notes that explained how the code actually worked (the implementation details) were the most critical. If the robot knew the specific steps the code was supposed to take, it could spot exactly where the robot went wrong.
The "What" Notes are Less Critical: Notes that just said "This sorts a list" were less helpful. The robot could often guess the "what" just by looking at the code or the function name.
The Danger of Bad Notes: The paper also tested what happens if you write notes based on the broken code. This was a disaster. The robot learned the wrong rules and got even more confused. It's like writing a manual for a broken car that tells you how to drive it into a wall.

The Bottom Line

The study concludes that we shouldn't throw away the "notes" (comments) when training AI to fix code. In fact, we should probably write better notes.

Think of it this way: If you want a mechanic (the AI) to fix your car, you don't just hand them the engine block; you give them the owner's manual too. The more clear and detailed that manual is, the better the mechanic can do their job. The authors suggest that developers should write clear comments not just for other humans, but to help these AI assistants do their best work.

Technical Summary: On the Impact of Code Comments for Automated Bug-Fixing

Problem Statement
Automated Bug Fixing (ABF) using Large Language Models (LLMs) is a critical application in Software Engineering. A prevalent preprocessing practice in ABF research, established by early works like Tufano et al. [56], involves removing code comments and abstracting identifiers to mitigate noise and Out-of-Vocabulary (OOV) issues, particularly when using Recurrent Neural Networks (RNNs). However, this study hypothesizes that code comments contain essential design and implementation insights necessary for understanding requirements and identifying bugs. The authors question whether the common practice of stripping comments during training and inference negatively impacts the bug-fixing capabilities of modern LLMs, which are better equipped to process natural language than their predecessors.

Methodology
The authors conducted an empirical study to evaluate the impact of code comments on ABF performance under four distinct conditions: training with/without comments and inference with/without comments.

Dataset Construction:
- The study utilized the dataset introduced by Tufano et al. [56], which contains buggy Java methods and their fixed versions mined from GitHub.
- Unlike the original work, the authors re-extracted instances from raw source code, preserving original identifiers and comments. They merged the original small (≤50 tokens) and medium (≤100 tokens) datasets and extended them to include larger methods (up to 512 tokens), resulting in a "Renewed" dataset ( $D_b$ ) of 116,372 pairs.
- Comment Generation: Since only ~20% of instances in $D_b$ contained developer-written comments (often of low quality), the authors used an automated procedure to generate high-quality, multi-intent comments. Using GPT-3.5 with in-context learning, they generated five specific categories of comments for the fixed version of each method: What (functionality), Why (rationale), How (implementation details), Usage (expected setup), and Property (pre/post-conditions). This created a comment-augmented dataset ( $D_c$ ).
- Validation: A manual analysis of 400 generated comments confirmed 93% coherence with their intended categories and a median adequacy score of 4/5.
Experimental Setup:
- Models: Two LLM families were fine-tuned: CodeT5+ (220M parameters, encoder-decoder) and DeepSeek-Coder (1.3B parameters, decoder-only).
- Training Conditions: Each model was fine-tuned on two datasets: one with generated comments ( $\Phi_c$ ) and one without ( $\Phi_b$ ).
- Inference Conditions: Models were tested on test sets containing comments ( $T_c$ ) and without comments ( $T_b$ ).
- Metrics: Performance was measured using Exact Match (EM) and CodeBLEU (CB). Statistical significance was assessed via McNemar's test and Odds Ratios (OR).
- Interpretability: To understand why comments matter, the authors employed SHAP (SHapley Additive exPlanations) using the GradientExplainer method to assign importance scores to input tokens (code vs. comments) and specific comment categories.

Key Contributions

Revisiting Preprocessing: The study challenges the standard practice of removing comments during ABF training, demonstrating that comments are not merely noise but valuable contextual signals.
Dataset Renovation: The authors provide a renovated, larger-scale dataset that retains raw source code and includes automatically generated, multi-intent comments, addressing the scarcity of high-quality commented bug-fix pairs.
Comprehensive Evaluation: The work provides the first controlled evaluation of LLMs across all combinations of training and inference conditions regarding comment presence.
Interpretability Analysis: The study offers granular insights into which types of comments (e.g., implementation details vs. functionality descriptions) contribute most to model reasoning.

Results

Impact of Comments (RQ1):
- Inference Time: The presence of comments at inference time significantly improves performance for all models.
- Training Time: Training with comments does not degrade performance when comments are absent at inference time (the effect size is negligible).
- Combined Effect: The highest performance gains occur when models are trained and tested with comments.
  - For CodeT5+, training and testing with comments increased Exact Match from 2.78% to 9.80% (a ~3x improvement).
  - For DeepSeek-Coder, the same configuration increased Exact Match from 5.09% to 12.77% (a ~2.5x improvement).
- Model Comparison: DeepSeek-Coder generally outperformed CodeT5+, likely due to its larger parameter count and architecture. However, CodeT5+ showed a more dramatic relative improvement (3x) when leveraging comments compared to DeepSeek-Coder (~2x), suggesting smaller models can close the performance gap with explicit contextual guidance.
Comment Importance (RQ2):
- SHAP analysis revealed that models consistently assign higher importance to code tokens than comment tokens overall, but comments still contribute significantly.
- Among comment categories, "How" comments (documenting implementation details) were the most influential for both models.
- "What" comments (describing functionality) were least important for CodeT5+ (likely redundant with method names), while "Property" and "Usage" comments were least important for DeepSeek-Coder.
Additional Findings:
- Comments generated from buggy code (rather than fixed code) did not help and slightly hindered performance, as they reinforced incorrect logic.
- Even without "How" comments, models still saw significant improvements over the baseline, indicating that non-implementation comments also provide value.

Significance and Claims
The paper claims that code comments are a critical, underutilized resource for Automated Bug Fixing. The authors assert that:

Documentation Benefits LLMs: Developers should document code not only for human maintainability but to enable LLMs to provide better assistance.
Training Practices: Researchers should reconsider removing comments from training datasets; doing so may discard valuable signals that improve model accuracy.
Quality Matters: While comments are beneficial, their quality and correctness are paramount. Comments generated from buggy code or those that are misleading can degrade performance.
Implementation Details are Key: Comments detailing how a method is implemented are the most effective at guiding models to fix bugs, likely because they highlight the specific logic that was violated in the buggy version.

The study concludes that while larger models possess inherent bug-fixing capabilities, the integration of high-quality, multi-intent comments can substantially enhance the performance of LLMs, potentially tripling accuracy in specific configurations.

Technical Summary: On the Impact of Code Comments for Automated Bug-Fixing

More like this