Technical Summary: AACR-Bench

Problem Statement

The deployment of Large Language Models (LLMs) for Automated Code Review (ACR) is hindered by the lack of high-quality, credible evaluation benchmarks. Existing benchmarks suffer from two critical limitations:

1. Incomplete Issue Annotation: Most datasets rely on raw Pull Request (PR) comments from GitHub as ground truth. This approach is inherently noisy and incomplete, as human reviewers often miss latent defects, leading to an underestimation of a model's ability to uncover potential issues.
2. Restricted Context Scope: Many code defects are cross-file, requiring repository-level context for accurate detection. While some existing benchmarks offer cross-file awareness, they are typically limited to a single programming language (e.g., Python). This lack of multi-language support restricts the generalizability of evaluation results and fails to represent the multilingual reality of modern software development.

Methodology

To address these challenges, the authors introduce AACR-Bench, a comprehensive benchmark designed for holistic, repository-level context evaluation.

Dataset Construction

• Scope: The dataset comprises 200 PRs and 1,505 fine-grained review comments extracted from 50 popular repositories across 10 mainstream programming languages (JavaScript, Python, TypeScript, Java, C#, C++, C, PHP, Go, and Rust).
• Annotation Pipeline: The authors employ an "AI-assisted, Human expert-verified" pipeline to overcome the incompleteness of raw PR data:
  1. Generation: Six mainstream LLMs (including Claude-4.5-Sonnet, Qwen3-Coder, GPT-5.2, DeepSeek-V3.2, GLM-4.7, and Gemini-3-Pro) generate review comments in parallel via two heterogeneous frameworks (an internal system and the open-source agent Claude Code).
  2. Augmentation: Original PR comments are augmented with deep semantic analysis of multi-turn review threads to extract confirmed defects.
  3. Human Verification: 80 senior software engineers (each with 2+ years of experience) rigorously verify the generated and augmented comments. This process involves double-blind annotation by a general pool, followed by adjudication by a six-member core expert team to resolve discrepancies.
  4. Context Labeling: Uniquely, the dataset annotates the required context level for each comment: Diff (local hunk), File (entire file), or Repo (repository-wide context including metadata and other files).
• Outcome: This process results in a 285% increase in issue coverage compared to datasets relying solely on raw PR comments.

Experimental Setup

The authors evaluate mainstream LLMs (both open-source and commercial) using AACR-Bench under varying conditions:

• Models: Qwen3-Coder-480B, DeepSeek-V3.2, GLM-4.7, GPT-5.2, and Claude-4.5-Sonnet.
• Context Retrieval Methods:
  • No Context: Baseline using only PR metadata.
  • BM25: Classic text similarity retrieval.
  • Embedding: Vector similarity retrieval (using Qwen3-Embedding-8B).
  • Agent-based: Autonomous context retrieval using the Claude Code framework.
• Metrics: Precision, Recall, and F1-score, analyzed across different context levels and programming languages.

Key Results

The evaluation reveals that previous assessments may have misjudged or only partially captured model capabilities due to data limitations. Key findings include:

1. Agent vs. Traditional Approaches: Agent-based methods exhibit a distinct trade-off. They achieve significantly higher Precision (e.g., Claude-4.5-Sonnet in Agent mode: 39.90% vs. 8.70% in No Context) but suffer from lower Recall (10.10% vs. 42.86%). This suggests agents excel at precise defect localization but may overlook potential issues due to "contextual tunnel vision." Conversely, traditional methods tend to generate more comments, increasing recall but introducing noise.
2. Context Retrieval Impact: The benefit of context retrieval is not universal and is highly model-dependent.
   • For some models (e.g., Claude-4.5-Sonnet), adding context via BM25 or Embedding actually degrades performance compared to the "No Context" baseline, likely due to noise introduction.
   • Other models (e.g., DeepSeek-V3.2) show improved performance with specific retrieval methods (BM25), while others (Qwen-480B-Coder) prefer Embedding.
3. Context Level Sensitivity:
   • Non-Agent methods generally show a performance decay as the required context level increases (Diff > File > Repo).
   • Agent-based methods often exhibit the opposite trend, performing better in complex Repo-level scenarios than in isolated Diff scenarios, though they may still miss obvious local issues.
4. Language-Specific Bias: Performance varies significantly across programming languages. For instance, C# and Java show robustness to context retrieval, while C and Rust often suffer performance drops when context is introduced. This suggests that intrinsic language characteristics (e.g., type systems, pointer management) and training data distribution heavily influence ACR performance.

Significance and Contributions

The paper claims the following contributions:

• AACR-Bench: The first multilingual, repository-level context-aware benchmark for LLM-enabled ACR tasks. It bridges the gap between single-language, diff-level datasets and the complex reality of production software.
• Improved Ground Truth: By combining multi-model generation with large-scale human annotation, the authors establish a more rigorous and comprehensive ground truth, increasing issue coverage by 285%.
• New Insights into ACR Capabilities: The evaluation challenges the assumption that "more context is always better." It demonstrates that the granularity of context and the choice of retrieval method significantly impact performance, with effects varying by LLM, programming language, and usage paradigm (Agent vs. traditional).
• Paradigm Shift: The work suggests a shift from "Passive Code Ingestion" (receiving pre-retrieved snippets) to "Active Code Auditing," where models must dynamically determine the necessity and granularity of retrieval to avoid noise and hallucination.

The authors conclude that while AACR-Bench provides a more rigorous standard, constructing a fully comprehensive ground truth remains a challenge due to the subjectivity of real-world software systems. Future work will focus on expanding the dataset scale and refining ground truth quality through advanced semi-automated methods.