🤖 AI

How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair

This paper presents the first empirical study demonstrating that successful LLM-based automated program repair relies on diffused attention across diverse diagnostic components in bug reports, whereas failures are caused by over-localized attention on metadata, highlighting attention misallocation as a key factor in repair inconsistency.

Original authors: Ramtin Ehsani, Irene Manotas, Saurabh Pujar, Luca Buratti, Preetha Chatterjee

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ramtin Ehsani, Irene Manotas, Saurabh Pujar, Luca Buratti, Preetha Chatterjee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Detective's Dilemma: Why AI Sometimes Misses the Clues

Imagine you are a detective trying to solve a mystery. You have a notebook full of clues: a witness's story, a blurry photo, a list of suspects, and a map of the crime scene. To solve the case, you need to read every page, connect the dots, and figure out which clue actually matters. Now, imagine you hire a super-smart robot detective to do the same job. You hand it the notebook, and it instantly writes a solution. But here's the weird part: sometimes the robot solves the mystery perfectly, and other times, it fails completely—even when you give it the exact same notebook!

This is the world of Large Language Models (LLMs) and Automated Program Repair. LLMs are like those super-smart robots; they are AI systems trained on massive amounts of text and code. They can write stories, answer questions, and even fix bugs (errors) in computer programs. "Automated Program Repair" is just a fancy term for asking the AI to look at a broken piece of software and a description of the problem, then write the code to fix it. But developers have noticed something frustrating: these AI detectives are inconsistent. They might fix one bug easily, but fail on a nearly identical one right next to it. Scientists want to know: Why? Is the AI just guessing, or is it looking at the wrong clues? This paper dives into the AI's "brain" to see exactly what it's paying attention to when it tries to fix a broken program.

The Paper's Big Discovery: Where the AI Looks Matters

In this study, the researchers decided to play a game of "spot the difference" with the AI's attention. They took 319 real-world bugs from popular software projects (written in Python and Java) and asked three different AI models to fix them. Some models were the big, expensive, closed-door ones (like the proprietary claude-4-sonnet), and others were open-source models (like gpt-oss-20b and qwen-3-32b).

To figure out what the AI was thinking, the researchers used a clever trick called perturbation analysis. Imagine you have a recipe for a cake, and you want to know which ingredient is the most important. You could try baking the cake without the flour, then without the sugar, and see which one ruins the cake the most. The researchers did the same thing with bug reports. They took a bug report—which usually has sections like "What went wrong," "How to make it break again," "What version of the software was used," and "What the code looks like"—and they secretly deleted one section at a time. Then, they asked the AI to try fixing the bug again. If the AI's fix changed a lot after deleting a section, that meant the AI was really paying attention to that part. If the fix stayed the same, the AI didn't care about that part.

The "Diffused" vs. "Localized" Attention Patterns

The researchers found two very different ways the AI looked at the clues, and these patterns told them everything about whether the fix would work.

1. The "Diffused" Detective (The Winner):
When the AI succeeded, it acted like a thorough detective. It spread its attention across many different parts of the bug report. It looked at the bug description (the story of what went wrong), the stacktrace (the technical error log that points to the exact line of code), and the test cases (examples of how the code should behave). The researchers call this diffused attention. It's like the AI was reading the whole notebook, connecting the witness's story to the map and the photo.

The Result: When the AI did this, it was much more likely to fix the bug. In fact, the study found that "diffused attention" was strongly linked to success.

2. The "Tunnel Vision" Detective (The Loser):
When the AI failed, it acted like a detective with tunnel vision. It got obsessed with one tiny, unimportant detail and ignored everything else. Often, it fixated on version information (like "Software Version 1.2.3" or "Operating System: Linux"). This is like a detective ignoring the murder weapon and the witness, and instead spending all their time staring at the suspect's shoe size.

The Result: When the AI focused too much on these boring metadata details, it usually failed to fix the bug. The study showed that "localized attention" (focusing on just one thing) was a strong sign that the repair would fail.

Do AI and Humans Agree on What's Important?

The researchers also wanted to know if the AI was looking at the same clues that human developers look at. To find out, they asked four experienced human developers to read 100 of the same bug reports and mark the parts they thought were most important.

The results were a mix of good news and bad news:

The Good News: When the AI did succeed, it was usually looking at the same sections the humans thought were important. The AI and the humans agreed on the top clues (like the bug description) about 54% of the time.
The Bad News: When the AI failed, it often ignored the human's top clues and focused on the wrong stuff (like version numbers). The study found that the more the AI's attention matched the human's attention, the higher the chance of a successful fix.

What the Paper Rules Out

It's important to note what this study didn't find. The researchers checked if the difficulty of the bug was the main reason for failure. They looked at "Easy," "Medium," and "Hard" bugs. They found that while harder bugs were indeed harder to fix, bug difficulty alone didn't explain why the AI failed. Even on easy bugs, the AI could fail if it had "tunnel vision" and ignored the right clues. This suggests that the problem isn't just that the bugs are too hard; the problem is that the AI is sometimes looking in the wrong place.

The Takeaway

This paper suggests that the secret to making AI better at fixing code isn't just giving it more data or making it smarter. It's about teaching it how to read. The study shows that successful repairs happen when the AI spreads its attention across the whole story—the symptoms, the error logs, and the expected behavior—rather than getting stuck on boring details like software versions.

By understanding that AI can suffer from "tunnel vision," developers can now design better instructions (prompts) to force the AI to look at the right clues. It's like teaching a detective to stop staring at the suspect's shoes and start looking at the murder weapon. The researchers have even created a new dataset of human-annotated bug reports to help train future AI models to pay attention to the right things, hopefully making them more reliable partners in fixing the software that runs our world.

Technical Summary: How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair

Problem Statement
Large Language Model (LLM)-based Automated Program Repair (APR) systems have demonstrated significant improvements over traditional techniques, yet their performance remains inconsistent. Models often succeed on one bug while failing on a nearly identical one, even when provided with the same contextual information. This unpredictability stems from a lack of interpretability regarding how LLMs process bug reports. Specifically, it is unclear how models prioritize diverse information within bug reports (e.g., bug descriptions, stacktraces, version metadata) and whether attention allocation patterns correlate with repair success. Prior studies have analyzed attention in code generation from short task descriptions, but extending this to real-world, heterogeneous bug reports containing both code and natural language remains an open challenge.

Methodology
The authors conducted an empirical study analyzing 319 real-world bugs from the SWE-bench Verified (Python) and Multi-SWE-bench (Java) datasets. The study utilized three LLMs: the proprietary claude-4-sonnet and open-source gpt-oss-20b and qwen-3-32b. To investigate attention mechanisms without requiring access to internal model states (which is often impossible with proprietary models), the authors employed perturbation-based analysis.

The methodology addressed three Research Questions (RQs):

RQ1 (Section-Level Attention): The authors treated bug reports as sets of six distinct sections (Bug description, Reproduction, Expected behavior, Actual behavior, Version information, and Additional information). They used Kernel SHAP (SHapley Additive exPlanations) to measure the marginal contribution of each section by masking them and observing changes in the generated patch's similarity to the ground truth (measured via CodeBLEU).
RQ2 (Fine-Grained Attention): Within each section, components were further broken down into natural language (e.g., descriptions, version info) and code (e.g., stacktraces, test snippets, class definitions). Using UniXcoder embeddings, the authors measured the semantic difference between the base output and outputs generated after masking individual components. This allowed for the classification of attention patterns into three structures: Diffused (attention spread across multiple components), Localized (focus on a single component), and No-attention.
RQ3 (Human-Model Alignment): To compare model attention with developer intuition, the authors created a new dataset of 100 manually annotated bug reports. Four experienced developers rated the importance of sections and identified key phrases. Alignment was measured using Hit@k metrics and Spearman rank correlations between developer ratings and model attention scores.

Key Results

Section-Level Differences: Successful repairs consistently allocated higher attention to the Bug description section, while unsuccessful repairs exhibited significantly higher attention to Version information (environmental metadata). This pattern held across all three models and both programming languages.
Attention Structures:
- Diffused Attention: Strongly associated with successful repairs (Odds Ratio [OR] = 2.07). Models that distributed attention across multiple diagnostic components (e.g., stacktraces, test data, and descriptions) were more likely to succeed.
- Localized Attention: Strongly associated with repair failures (OR = 0.40). Over-focusing on a single component, particularly non-diagnostic metadata, often led to incorrect patches.
- Component Importance: Attention to Code: Stacktrace and NL: Description showed significant positive associations with success. Conversely, attention to NL: Version information was negatively associated with success.
Human-Model Alignment: There is a statistically significant correlation between successful repairs and the alignment of model attention with developer-identified important sections and phrases. Successful repairs showed higher Spearman correlations ( $\rho=0.60$ ) and Hit@1 scores compared to failures. Misalignment, where models focused on version numbers while developers focused on failure descriptions, was a common cause of failure.

Significance and Contributions
The paper claims to provide the first empirical study of attention patterns in LLM-based APR. Its primary contributions are:

Empirical Evidence of Attention Misallocation: The study establishes that attention misallocation—specifically the over-localization on metadata and under-utilization of diagnostic components—is a key factor in APR failures.
Characterization of Attention Patterns: It characterizes the distinction between "diffused" (successful) and "localized" (often failing) attention strategies across section and component levels.
Human-Model Alignment Metric: It demonstrates that repair success is linked to how well model attention aligns with developer reasoning, offering a new metric for evaluating APR reliability.
New Dataset: The authors release the first annotated developer attention dataset for APR, comprising 100 bug reports with section-level importance ratings and key phrase annotations.

Conclusion
The authors conclude that while bug difficulty influences repair rates, it does not fully explain performance variance. Instead, the distribution of attention is a critical determinant. Successful repairs rely on integrating signals from multiple diagnostic sources, whereas failures often result from models fixating on less informative metadata. These findings suggest that future APR systems should prioritize mechanisms that encourage diffused attention and align model information prioritization with developer intuition, potentially through improved prompt engineering, context selection, or fine-tuning using the provided attention dataset.