Technical Summary: Code-QA-Bench

Problem Statement

Current benchmarks for AI coding agents, such as HumanEval, MBPP, and SWE-Bench, primarily focus on code generation or issue resolution. While effective for measuring patch generation, they fail to adequately evaluate code comprehension—the ability to understand existing codebases, trace control flow, locate features, and explain behavior. A critical gap exists in distinguishing genuine code reasoning from documentation recall or pretraining memorization. Existing repository-level QA benchmarks often evaluate agents on repositories with full documentation intact, making it difficult to determine if an agent is reading the code or simply recalling information from its training data or the provided documentation.

Methodology

Code-QA-Bench introduces a fully automated framework designed to synthesize repository-level QA benchmarks that isolate code understanding from documentation and memorization. The framework operates on four core principles:

1. Three-Condition Experimental Design

To quantify the specific contributions of code access, documentation, and memorization, every task is evaluated under three distinct conditions:

• Closed-book: The agent receives only the question with no repository access. This measures prior knowledge and memorization.
• Code-only: The agent has access to the repository with all natural-language content removed (docstrings, comments, READMEs, and documentation files). This measures pure code-structural reasoning.
• Documented: The agent has access to the full repository including all documentation. This measures code reasoning augmented by documentation.

Key metrics are derived from the deltas between these conditions:

• Code-only - Closed-book: The genuine contribution of code reading beyond memorization.
• Documented - Code-only: The utility of documentation for code comprehension.

2. Documentation Removal (Environment-Level Control)

To force code-structural reasoning, the framework creates a "code-only" version of each repository by programmatically stripping:

• Docstrings: Identified via AST and removed (with pass inserted to maintain syntax).
• Comments: Full-line and inline comments are removed or truncated.
• Documentation Files: Directories like docs/, doc/, and files like README*, *.md, and *.rst are deleted.
  Crucially, executable code, imports, type annotations, and string literals are preserved, ensuring semantic signals from identifiers remain.

3. Answer-First Task Generation

Unlike previous approaches that generate questions first, Code-QA-Bench employs an answer-first pipeline:

1. Chunk Selection: Documentation chunks are extracted and scored based on content quality, code references, and structural signals.
2. Gold Answer Generation: A tool-equipped agent explores the source code (using read_file, list_directory, search_code) to produce a verified "gold answer." The agent is required to trace at least one level deeper than the documentation and include facts not present in the text.
3. Verification: A "code-only audit" ensures the gold answer contains no claims that can only be recovered from documentation. If a claim is documentation-dependent, it is removed or rewritten.
4. Question Derivation: A natural language question is derived from the verified gold answer, ensuring the task is grounded in real code structure.

4. Dual Task Set

The benchmark generates two distinct task sets:

• Code-Derivable Tasks (528 tasks): Gold answers are verified to be recoverable from code structure alone. These tasks validate the design (expecting Code-only ≈ Documented).
• Doc-Dependent Tasks (100 tasks): Gold answers are generated from documentation only, intentionally requiring documentation to answer fully. These tasks quantify documentation utility (expecting Documented > Code-only).

5. Evaluation

Tasks are scored by an LLM judge (GPT-5.4) on a 0–5 scale across three axes:

• Accuracy: Correctness of factual claims.
• Completeness: Coverage of key points in the rubric.
• Specificity: Reference to specific files, functions, or code patterns.
  The final score is the normalized mean of these three axes.

Key Results

Experiments were conducted on four frontier models (Claude Opus 4.6, DeepSeek-V4-Pro, Kimi-K2.6, Gemini-3.1-Pro) across 10 Python repositories from SWE-Bench.

1. Code Access is the Dominant Factor

Access to the codebase provides a substantial performance gain over memorization. The mean gain from Code-only over Closed-book is +0.23, which is three times larger than the gain provided by documentation. This confirms that reading code contributes significantly more to understanding than prior knowledge alone.

2. Documentation Provides Modest, Measurable Utility

For doc-dependent tasks, access to documentation yields a consistent, statistically significant improvement (Documented - Code-only = +0.071, p < 0.003). This suggests that while code structure allows agents to infer much of the answer, documentation provides critical details (design rationale, edge cases) that improve completeness and accuracy.

3. Validation of Experimental Design

On code-derivable tasks, the performance gap between Code-only and Documented conditions is negligible (∆ ≈ +0.007), and statistically insignificant for most models. This validates the methodology: the framework successfully isolates tasks where documentation is unnecessary, proving that the observed benefits on doc-dependent tasks are genuine utility rather than artifacts of the evaluation setup.

4. Model-Specific Insights

• Memorization: Models score 0.56–0.68 in the closed-book condition, indicating significant pretraining memorization of well-known libraries.
• Reasoning vs. Recall: DeepSeek-V4-Pro showed the largest gap between code-only and closed-book (+0.450), suggesting a heavy reliance on active code exploration rather than parametric recall.
• Category Analysis: Contrary to the hypothesis that "Why" questions would show the largest documentation gap, "Where" questions (feature location) showed the most significant delta on code-derivable tasks, suggesting documentation acts as a navigation index.

Significance and Claims

The paper claims that Code-QA-Bench provides a necessary methodological shift in evaluating AI coding agents by:

1. Separating Comprehension from Memorization: It offers a quantitative way to measure how much an agent relies on reading code versus recalling training data or documentation.
2. Environment-Level Control: By removing documentation at the repository level rather than filtering questions, it forces agents to engage with code structure, addressing the "contamination" problem in existing benchmarks.
3. Automated and Reproducible: The pipeline is fully automated, repo-agnostic, and applicable to any well-documented Python repository, allowing for continuous benchmark updates as models improve.
4. Diagnostic Value: The three-condition design provides diagnostic signals (e.g., specificity saturation, memorization levels) that generation-only benchmarks miss, offering a more nuanced view of an agent's capabilities.

The authors conclude that while frontier models are strong code-structural readers, the modest but consistent benefit of documentation highlights the continued importance of well-documented codebases for AI agents. The framework is open-source and serves as both an evaluation tool and a source of verified training data.