SWE-QA: Can Language Models Answer Repository-level Code Questions?

1. Problem Statement

Current Large Language Models (LLMs) have shown promise in code understanding, but existing benchmarks (e.g., CoSQA, CodeQA) predominantly focus on isolated code snippets, functions, or APIs. These setups fail to capture the complexity of real-world software engineering, where developers must:

• Navigate large, interconnected codebases spanning multiple files.
• Understand software architecture and design rationales.
• Trace long-range dependencies and perform multi-hop reasoning.
• Synthesize architectural knowledge to answer complex "Why" and "How" questions.

There is a lack of high-quality, human-verified benchmarks that evaluate LLMs on repository-level question answering (QA) requiring deep, multi-file context and structural understanding.

2. Methodology: SWE-QA Benchmark Construction

The authors propose SWE-QA, a repository-level code QA benchmark constructed through a four-stage pipeline (Figure 2) involving 15 diverse Python repositories (12 from SWE-Bench, 3 from SWE-Bench-Live).

A. Seed Collection & Taxonomy Construction

• Data Source: Crawled 77,100 GitHub issues from 12 popular repositories.
• Filtering: Selected 41,955 issues with substantial content (>1,000 chars) and used an LLM to extract 127,415 explicit code-related questions.
• Taxonomy: Two human experts with 3+ years of experience manually analyzed 1,000 sampled questions to create a two-level taxonomy:
  • Level 1 (Interrogative): What, Why, Where, How.
  • Level 2 (Intent): 12 fine-grained categories (e.g., Dependency Tracing, Design Rationale, Feature Location, Algorithm Implementation).
• Distribution: "How" questions (35.2%) and "Where" questions (28.4%) are most frequent, reflecting the need for procedural and locational knowledge.

B. Question Instantiation

• Context Extraction: Used tree-sitter to parse repository structures into typed graphs (nodes: classes, methods, files; edges: calls, imports).
• Generation: Selected focal elements (e.g., a specific class) and combined their structural context with seed templates derived from the taxonomy.
• Process: An LLM generated candidate questions tailored to specific repositories, ensuring they required multi-hop reasoning and could not be answered by simple retrieval.

C. Answer Collection & Validation

• RAG Pipeline: Answers were generated using a Retrieval-Augmented Generation (RAG) approach, retrieving relevant code snippets, docs, and metadata.
• Human-in-the-Loop:
  • Expert Revision: Two senior developers reviewed each answer for factual accuracy, completeness, and clarity using tools like Cursor. Disagreements were resolved by a third expert.
  • Quality Filtering: Pairs were discarded if they were ambiguous, factually incorrect, or lacked sufficient grounding in the code.
• Final Dataset: 720 high-quality QA pairs (48 per repository), covering 13,300 files and over 3.4 million lines of code.

3. Key Contributions

1. SWE-QA Benchmark: A repository-level QA benchmark with 720 human-verified QA pairs spanning 15 Python repositories. It uniquely combines multi-hop reasoning, cross-file dependencies, and architectural understanding.
2. Flexible Generation Pipeline: A modular framework that allows for the semi-automated creation of QA datasets for any new open-source repository using seed templates and static code analysis.
3. Comprehensive Evaluation: A rigorous assessment of LLMs under various context augmentation strategies (Direct Prompting, RAG, Agent Frameworks).

4. Experimental Results

The authors evaluated six advanced LLMs (including GPT-5.1, Gemini 2.5 Pro, GLM-4.6, and Qwen3-Coder) using different strategies:

• Context Augmentation Impact:
  • Direct Prompting: Performed poorly (e.g., Kimi K2 scored ~51/100), highlighting the necessity of repository context.
  • RAG (Sliding Window/Function Chunking): Significantly improved performance (e.g., +10–14 points).
  • Agent Frameworks (OpenHands, SWE-agent): Achieved the best results by enabling iterative reasoning and tool use. OpenHands with GPT-5.1 achieved the highest score of 70.79/100.
• Model Performance:
  • GPT-5.1 and GLM-4.6 (with OpenHands) led the pack.
  • Smaller models (e.g., Qwen3-30B) struggled with agent frameworks, performing similarly to or worse than RAG, suggesting limitations in long-term memory and tool planning.
  • Commercial tools (Cursor, Tongyi Lingma) performed competitively (69–70), validating end-to-end tool-augmented solutions.
• Taxonomy Analysis:
  • Models excelled at "Why" questions (Design Rationale) and "How" questions regarding API support, where information is often explicit in docstrings.
  • Models struggled with "What" (Architecture exploration) and "Where" (Feature location) questions, which require reconstructing dispersed logic across files.
• Cross-Repository Generalization:
  • Performance varied by repository complexity. "Flask" was easiest (75.42), while "Pylint" (62.01) and "Conan" (64.51) were most challenging.
  • Repositories from SWE-Bench-Live (less data leakage) were harder than those from SWE-Bench.

5. Significance and Future Directions

• Realistic Evaluation: SWE-QA moves beyond snippet-level testing to evaluate the "whole repository" understanding required for real-world software engineering tasks.
• Agent Frameworks: The results suggest that while RAG helps, Agent-based frameworks are currently the most effective approach for navigating complex, multi-file codebases, despite higher token costs.
• Limitations: The benchmark is currently Python-only and based on static snapshots. Future work aims to extend to other languages and dynamic, evolving repositories.
• Open Challenges: The paper highlights that LLMs still struggle with deep dependency tracing and precise locational reasoning, indicating a need for better code graph integration and reasoning capabilities.

The paper concludes that while LLMs show promise in repository-level QA, particularly when augmented with agents, significant challenges remain in handling complex, multi-hop dependencies and architectural reasoning.