🤖 AI

Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering

This paper introduces StackRepoQA, the first repository-level question answering dataset derived from 1,318 real developer questions across 134 Java projects, and uses it to demonstrate that while retrieval-augmented methods improve LLM performance, current models still struggle with genuine repository-scale comprehension, often relying on memorized Stack Overflow answers rather than true reasoning.

Original authors: Yoseph Berhanu Alebachew, Hunter Leary, Swanand Vaishampayan, Chris Brown

Published 2026-03-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yoseph Berhanu Alebachew, Hunter Leary, Swanand Vaishampayan, Chris Brown

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to fix a massive, 100-year-old library. You don't just need to know how to read a single book; you need to understand how Book A in the History section connects to a map in the Geography section, which then links to a diary in the Biography wing.

This is the challenge of Repository-Level Question Answering. It's about asking a computer to understand an entire software project (a "repository") rather than just a tiny snippet of code.

Here is a breakdown of the paper "Beyond Code Snippets" using simple analogies.

1. The Problem: The "Single-Page" Trap

Most current AI models (Large Language Models or LLMs) are like brilliant students who only study flashcards. They are amazing at answering questions about a single sentence or a short paragraph of code.

But real-world software is like a giant, interconnected city. To fix a traffic light, you might need to know about the power grid, the city planning laws, and the traffic sensors.

The Issue: When developers ask AI questions about their whole project, the AI often gets lost because it's used to looking at just one "flashcard" (a single file) instead of the whole "city map."

2. The New Tool: StackRepoQA

The researchers built a new testing ground called StackRepoQA.

The Analogy: Imagine they took 1,318 real questions that developers asked on a giant forum (Stack Overflow) and matched them to the actual "blueprints" (GitHub repositories) of 134 different software projects.
Why it matters: Before this, we only tested AI on isolated puzzles. Now, we are testing them on the real, messy, multi-file projects they will actually face in the real world.

3. The Experiment: Memory vs. Reasoning

The researchers asked two questions:

Can the AI answer these questions on its own?
Does giving the AI a "search tool" (Retrieval-Augmented Generation or RAG) help?

They tested two main strategies for the "search tool":

File-Level Search (The "Keyword" Search): Like searching a library by typing a word into a catalog. It finds books with that word, but maybe not the right chapter.
Graph-Based Search (The "Map" Search): Like having a librarian who understands the structure of the library. They know that "Chapter 3" depends on "Chapter 1," even if the words are different. This is Graph RAG.

4. The Big Surprise: The "Cheat Sheet" Effect

Here is the most shocking finding: The AI was mostly cheating.

When the AI answered questions without any help, it got a decent score (around 6 out of 10). But when the researchers asked: "Hey, don't use your memory. Only look at the files we just gave you," the AI's score plummeted.

The Metaphor: It turns out the AI had memorized the answers to these specific questions because they were in its training data (like a student who memorized the answer key to a practice test). It wasn't actually "thinking" or "reasoning" through the code; it was just reciting what it had seen before.
The Proof: When they tested questions that were asked after the AI was trained (new questions), the AI got much worse. This proved that its earlier success was just memorization, not true understanding.

5. The Solution: The "Architect" vs. The "Librarian"

When they forced the AI to rely on the "search tools":

The "Keyword" Search (File-RAG): Helped a little bit. It was like giving the AI a stack of random pages. It found some relevant info but got distracted by noise.
The "Map" Search (Graph-RAG): This was the winner. By understanding the structure (how classes connect, who calls whom), the AI performed significantly better.
- Analogy: If you ask a human, "How do I fix the engine?" and they just read a manual page about "oil," they might miss the point. But if they have a diagram showing how the oil pump connects to the engine block, they can actually solve the problem. The Graph RAG provided that diagram.

6. The Takeaway for the Real World

The paper concludes with a warning and a hope:

The Warning: We cannot just trust AI to fix our software projects right now. If the AI hasn't seen the project before, or if the project is private (not on the internet), the AI might hallucinate or give outdated advice. It's currently too reliant on "cheat sheets" (memorized data) rather than genuine reasoning.
The Hope: If we give AI the right tools—specifically structural maps (Graph RAG) that show how code pieces fit together—it can get much better at understanding complex systems.

In a nutshell:
Current AI is like a student who aces the practice test by memorizing answers but fails the real exam when the questions are slightly different. To make AI a reliable engineer, we need to stop testing it on flashcards and start giving it blueprints and maps so it can learn to actually think about how the whole system works.

1. Problem Statement

Current benchmarks for Large Language Models (LLMs) in software engineering predominantly focus on isolated code snippets (single functions or files). This narrow scope fails to capture the complexity of real-world program comprehension, which often requires reasoning across multiple files, modules, and system-level dependencies.

Existing repository-level datasets suffer from significant limitations:

Synthetic Data: Many datasets use LLM-generated questions from GitHub issues, which may not reflect authentic developer intent or include verified solutions.
Lack of Temporal Control: Most benchmarks do not account for the model's training cutoff dates, making it difficult to distinguish between genuine reasoning and the memorization of training data (e.g., Stack Overflow answers).
Retval Limitations: Simple retrieval-augmented generation (RAG) based on flat text similarity often fails to capture the structural relationships (e.g., call graphs, inheritance) inherent in codebases.

The authors aim to address these gaps by creating a rigorous benchmark to evaluate how well LLMs comprehend entire repositories and whether retrieval strategies can mitigate the limitations of context windows and training data exposure.

2. Methodology

A. Dataset Construction: StackRepoQA

The authors introduce StackRepoQA, the first multi-project, repository-level QA dataset derived from real developer interactions.

Source: 1,318 real developer questions with accepted answers from Stack Overflow.
Scope: Mapped to 134 actively maintained open-source Java projects on GitHub.
Selection Criteria: Repositories were filtered for size (>10MB), activity (commits in last 12 months), community engagement (stars, contributors), and license type.
Mapping Strategy: Questions were mapped to repositories via keyword search on repository names. To ensure quality, a senior Java developer manually reviewed questions to identify false positives (unrelated questions).
Unique Feature: The dataset includes temporal cutoffs. Questions are categorized as "pre-cutoff" (likely in training data) or "post-cutoff" (unseen by the model), enabling the disentanglement of memorization from reasoning.

B. Experimental Setup

The study evaluated two state-of-the-art LLMs: Claude 3.5 Sonnet and GPT-4o.

Baseline: Direct prompting without external tools.
Agentic RAG Framework: A multi-agent system built using AutoGen was implemented to test augmentation strategies:
1. File-RAG Agent: Retrieves top- $k$ semantically relevant files using a vector store (ChromaDB) and cosine similarity.
2. Graph-RAG Agent: Queries a Neo4j graph database constructed from the repository's Abstract Syntax Tree (AST) and static dependencies (calls, extends, uses). This agent generates Cypher queries to retrieve structural subgraphs.
3. Summarization & Supervisor Agents: Coordinate the flow, integrate retrieved context, and synthesize final answers.
Evaluation Strategy:
- LLM-as-a-Judge: Responses were scored 1–10 by LLMs based on correctness, completeness, and faithfulness.
- Human Validation: Two senior engineers validated a subset of responses, confirming high agreement with the LLM judges (Cohen's $\kappa \approx 0.78-0.87$ ).
- Ablation Study: Tested configurations: LLM only, LLM + File-RAG, LLM + Graph-RAG, and LLM + Both.

3. Key Contributions

StackRepoQA Dataset: A publicly available, multi-project benchmark containing 1,318 real Q&A pairs mapped to 134 Java repositories, designed to test repository-scale comprehension.
Systematic Evaluation of RAG: A comparative analysis of file-level vs. graph-based retrieval strategies within a multi-agent framework.
Memorization vs. Reasoning Analysis: The first empirical study to use temporal cutoffs to prove that high baseline scores on repository QA are often driven by memorization of Stack Overflow content rather than code reasoning.
Structural Insights: Evidence that graph-based structural retrieval significantly outperforms flat text retrieval for complex code comprehension tasks.

4. Key Results

RQ1: Baseline Accuracy & Memorization

Moderate Baseline: Without augmentation, LLMs achieved moderate accuracy (average scores ~6.7–6.9).
Memorization Dominance: Performance dropped significantly when models were instructed to ignore internal knowledge and rely only on retrieved context.
Cutoff Effect:
- Pre-cutoff questions: High scores (e.g., Claude: 6.59, GPT-4o: 6.97).
- Post-cutoff questions: Sharp decline (e.g., Claude: 5.44, GPT-4o: 5.97).
- Conclusion: Much of the "success" on public repositories is due to the models having seen the specific Stack Overflow Q&A pairs during pretraining.

RQ2: Impact of Augmentation

Graph-RAG Superiority: Graph-based retrieval yielded the highest performance gains.
- Graph-RAG only: Score 6.12 (related questions).
- File-RAG only: Score 5.66.
- Combined: Score 6.11 (when ignoring internal knowledge).
The "Distracting Effect": For unrelated questions, adding retrieval context actually degraded performance, confirming that irrelevant context can confuse LLMs.
Unseen Data: Even with Graph-RAG, performance on post-cutoff (unseen) questions remained modest (~5.68), indicating that current LLMs still struggle with genuine reasoning on new codebases.

5. Significance and Implications

Re-evaluating Benchmarks: The study warns that current benchmarks may overstate LLM capabilities due to data contamination (memorization). Future evaluations must use temporal cutoffs and unseen repositories to measure true reasoning.
Importance of Structure: Flat text retrieval is insufficient for repository-level tasks. Graph-based representations (capturing dependencies, inheritance, and call hierarchies) are critical for grounding LLMs in code structure.
Practical Limitations: Practitioners should not rely on LLMs as standalone solutions for private or rapidly evolving codebases where training data exposure is low. Structured retrieval mechanisms are essential for reliable program comprehension.
Future Directions: The authors advocate for benchmarks that explicitly separate memorization from reasoning and suggest exploring richer behavioral signals (data flow, execution traces) and visual representations (call graphs) to further improve comprehension.

In summary, StackRepoQA establishes a new standard for evaluating LLMs in software engineering, revealing that while LLMs have memorized vast amounts of public code knowledge, their ability to reason over unseen, complex repository structures remains limited without sophisticated, structure-aware retrieval systems.