💻 computer science

Hierarchical Knowledge Injection for Improving LLM-based Program Repair

This paper proposes a layered knowledge injection framework that incrementally augments LLMs with bug, repository, and project-level context to improve automated program repair, achieving a significant increase in fix rates across various bug types.

Original authors: Ramtin Ehsani, Esteban Parra, Sonia Haiduc, Preetha Chatterjee

Published 2026-02-10

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Ramtin Ehsani, Esteban Parra, Sonia Haiduc, Preetha Chatterjee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery in a massive, sprawling mansion.

Currently, most AI "detectives" (Large Language Models) are being sent into a room with a broken vase, a single note from a witness, and a photo of the broken pieces. They try to figure out who broke it and how to fix it using only what is in that one room. Sometimes they succeed, but often they fail because they don't know that the person who broke the vase was actually running from a ghost in the hallway, or that the vase was part of a set that requires a specific type of glue found only in the basement.

This paper, "Hierarchical Knowledge Injection for Improving LLM-based Program Repair," proposes a better way to train these AI detectives. Instead of just throwing them into a room, the researchers give them a "Layered Briefing."

The Three Layers of the Briefing

Think of the researchers' method like a detective getting more and more information as the case gets tougher:

Layer 1: The "Crime Scene" Knowledge (Bug Knowledge)
This is the basic stuff. The AI gets the broken code (the broken vase), the error message (the witness statement), and the failed test (the evidence). This works for "simple" crimes—like someone accidentally tripping over the vase.

Layer 2: The "Floor Plan" Knowledge (Repository Knowledge)
If the AI can't solve it with just the crime scene, the researchers give it the blueprints of the house. Now the AI knows which doors lead where, which rooms are connected, and who was moving through the hallways recently (the history of the code). This helps the AI realize, "Oh, the vase didn't just fall; it was pushed by a change made in the kitchen next door!"

Layer 3: The "Family History" Knowledge (Project Knowledge)
If it’s still not solved, the AI gets the family archives. This includes the "house rules" (documentation) and stories of how previous broken items were fixed in the past (previous bug fixes). This is crucial for complex mysteries, like realizing, "In this house, we always use silver polish for all glassware, even if it looks like ceramic."

Why does this matter? (The Results)

The researchers tested this on a large set of coding "mysteries" using two different AI models (Llama 3.3 and GPT-4o-mini). Here is what they found:

It actually works: By adding these layers, the AI's success rate jumped significantly. One model went from a 56% success rate to a 79% success rate. That is a massive leap in the world of AI.
Don't dump everything at once: Interestingly, if you give the AI all the information (the crime scene, the blueprints, and the archives) at the very beginning, it actually gets confused. It’s like giving a detective a 500-page book of history when they just need to know where the hammer is. The "layered" approach—giving info only when needed—is much smarter.
Some mysteries are still too hard: Even with all the info, some "crimes" (like complex network errors or visual/GUI bugs) are still too tricky. These are like "ghost stories"—they require a level of intuition and real-time interaction that current AI just hasn't mastered yet.

The Big Picture

The paper concludes that if we want AI to help humans write and fix software, we shouldn't just give it the "broken part." We need to give it the context—the history, the connections, and the rules of the world it is working in. We are moving from "AI that looks at code" to "AI that understands the whole system."

Technical Summary: Hierarchical Knowledge Injection for Improving LLM-based Program Repair

1. Problem Statement

While Large Language Models (LLMs) have revolutionized Automated Program Repair (APR) by generating patches with minimal supervision, they frequently fail to resolve complex bugs. The core limitation identified is a lack of contextual awareness. Most existing LLM-based APR methods rely solely on "local" context (the buggy function, error messages, and stack traces). However, real-world software bugs often stem from dependencies, project-level constraints, documentation requirements, or historical patterns that exist outside the immediate scope of the faulty code. When LLMs lack this broader understanding, they often produce "hallucinated" fixes or patches that violate project-specific logic.

2. Methodology

The authors propose a Layered Knowledge Injection Framework that incrementally augments the LLM's prompt with structured information. Instead of providing all information at once (which can cause noise and performance degradation), the framework follows a hierarchical approach:

Layer 1: Bug Knowledge Layer (Local Context): Provides immediate facts such as the buggy function, failing test cases, error messages/stack traces, runtime/angelic variables (expected values), and GitHub issue descriptions.
Layer 2: Repository Knowledge Layer (Structural Context): For bugs unresolved in Layer 1, the framework injects repository-level data, including co-occurring files (files frequently committed together), structural dependencies (definitions of invoked classes/functions), and recent commit history.
Layer 3: Project Knowledge Layer (Semantic Context): For remaining unresolved bugs, the framework injects high-level insights extracted via dense retrieval (FAISS) from official project documentation and the history of previously resolved issues.

Evaluation Setup:

Dataset: 314 bug-patch pairs from the BugsInPy benchmark, manually categorized into six bug types (Program Anomaly, Network, Configuration, GUI-Related, Performance, and Permission/Deprecation).
Models: Llama 3.3 and GPT-4o-mini.
Metrics: Pass@k (probability of at least one successful fix in $k$ attempts) and the total number of fixed bugs (#fixed).

3. Key Contributions

Hierarchical Framework: A novel, structured method for progressive context injection that scales from local to project-wide knowledge.
Bug-Type Specific Analysis: A detailed investigation into how different categories of bugs (e.g., GUI vs. Network) respond to different levels of context.
Comparative Benchmarking: Evaluation against both a prior state-of-the-art fact-selection approach and an "all-at-once" baseline, proving that incremental injection is superior to flooding the prompt with all available data.
Complexity & Error Analysis: A statistical analysis (using Mann-Whitney U tests and Cohen’s $d$ ) linking unresolved bugs to higher code complexity (Cyclomatic Complexity, LOC) and structural isolation.

4. Results

Significant Performance Gains: Using Llama 3.3, the framework achieved a 79% fix rate, representing a 23% improvement over previous work.
Layered Effectiveness:
- Repository Layer: Provided a significant boost (approx. 9%) across all bug types, proving that structural context is universally beneficial.
- Project Layer: Provided targeted benefits, specifically helping resolve Program Anomaly, GUI, and Network bugs, but offering less help for performance-related issues.
The "All-at-Once" Failure: Providing all layers of knowledge simultaneously actually decreased performance (Llama 3.3 dropped to 65% fix rate), confirming that excessive, unfiltered context acts as noise.
Remaining Challenges: Unresolved bugs were found to be significantly more complex (higher Lines of Code and Cyclomatic Complexity) and often required reasoning about user-facing behavior or implicit runtime conditions that static prompts cannot capture.

5. Significance

This research shifts the focus of LLM-based APR from "better prompting" to "better knowledge engineering." It demonstrates that the bottleneck in automated repair is not just the reasoning capability of the LLM, but the strategic retrieval and organization of software ecosystem knowledge. The findings suggest that the next generation of APR tools should be adaptive and agentic—systems that don't just read a prompt, but actively navigate a repository and interact with a runtime environment to gather the specific context required for the task at hand.