Technical Summary: Bigger Isn't Always Better: A Comparative Evaluation of LLMs for Automated Code Review

Problem Statement

Automated code review using Large Language Models (LLMs) has become integral to modern software development toolchains (e.g., CodeRabbit, GitHub Copilot). A critical engineering decision in these systems is model selection: determining which LLM provides the highest-quality reviews at acceptable cost and latency. Conventional wisdom suggests that larger, more expensive models should outperform smaller ones on complex reasoning tasks like code review, which requires understanding control flow, data dependencies, and domain conventions. This paper empirically tests that assumption and simultaneously critiques the prevailing evaluation methodology, which relies heavily on synthetic benchmarks that may not reflect real-world performance.

Methodology

Evaluation Framework

The authors propose a reusable two-pass evaluation framework to address the challenge of matching free-form model findings against ground-truth annotations:

1. Pass 1 (Deterministic Matching): Model findings are matched to ground truth using three criteria: normalized file path, line range overlap (±5 lines), and comment type compatibility. This pass handles approximately 70% of cases (clear true positives, false positives, and false negatives) at zero LLM cost.
2. Pass 2 (LLM Adjudication): Ambiguous cases are adjudicated by Claude Opus 4.6 (temperature 0.0) using a structured rubric. The judge classifies findings as TRUE_POSITIVE, FALSE_POSITIVE, or PARTIAL_MATCH and scores four qualitative dimensions: Depth, Context Awareness, Specificity, and Suggestion Correctness.

Models and Dataset

• Models Evaluated: Five models across three providers with a 15× cost range:
  • Anthropic: Claude Sonnet 4.6 (large), Claude Haiku 4.5 (small).
  • OpenAI: GPT-5.4 mini.
  • Minimax: Minimax M2.7.
  • Zhipu AI: GLM-5 Turbo.
• Dataset (150 Samples):
  • Synthetic (n=100): 13 mutation operators injecting known bugs (e.g., SQL injection, off-by-one errors) into 8 code templates (TypeScript, Python, Go). These are small (median 5 lines).
  • Real (n=50): Merged bug-fix pull requests from eight major repositories (e.g., vercel/next.js, facebook/react). These are substantially larger (median 117 lines) and contain multi-file changes and formatting noise.

Experimental Conditions

Three experiments were conducted to test scalability and generalization:

1. Calibration (n=25): Synthetic only.
2. Stable Baseline (n=100): Synthetic only.
3. Generalization (n=150): 100 synthetic + 50 real PRs.

Key Results

1. Smaller Models Outperform Larger Ones

Contrary to the "larger-is-better" assumption, Claude Haiku 4.5 consistently outperformed the larger Claude Sonnet 4.6 across all experimental conditions.

• Performance: Haiku achieved a higher F1 score (0.365 vs. 0.343) and significantly higher recall (0.293 vs. 0.248, an 18% increase) on the full dataset.
• Qualitative Scores: Haiku scored higher on all four qualitative dimensions (Depth, Context, Specificity, Suggestions).
• Cost Efficiency: Haiku operated at 3.2× lower cost per review than Sonnet.
• External Validation: These results were replicated on the independent Martian Code Review Benchmark, where Haiku placed higher than Sonnet (F1 36.4% vs. 27.1%).

2. The Synthetic-to-Real Gap

The paper identifies a severe degradation in model performance when moving from synthetic to real-world data.

• Magnitude: On synthetic samples alone, the best model (Haiku) achieved an F1 of 0.847. On real PRs alone, this dropped to 0.066 (a 92% degradation).
• Cause: The degradation is driven by diff size, formatting noise, and multi-file interactions absent in synthetic data.
• Diff Size Predictor: Diff size is the dominant predictor of performance. F1 drops from ~0.66–0.80 on small diffs (<10 lines) to ~0.04–0.07 on large diffs (>150 lines).

3. Category and Language Analysis

• Bug Categories: Security bugs are well-detected (~70% recall). Logic bugs show divergence (Haiku 24.5% vs. Sonnet 19.6%). Performance bugs (e.g., N+1 queries) are largely undetectable, with four of five models achieving 0% recall.
• Languages: Go reviews yielded the best results (Haiku F1 0.485), likely due to Go's explicit error handling and stronger typing, compared to TypeScript (0.272) and Python (0.209).

4. Ensemble Strategies

Combining models (taking the union of findings from Haiku and another model) decreased overall F1 scores. The models largely detected the same bugs; adding a second model introduced false positives without meaningfully increasing true positives.

Significance and Claims

The paper claims three primary contributions to the field of automated code review:

1. Challenging the Scale Paradigm: It provides empirical evidence that for code review, a smaller, cheaper model (Haiku 4.5) can outperform a larger, more expensive one (Sonnet 4.6) in both quality and cost-efficiency. This suggests that code review may not benefit from additional model scale beyond a certain capability threshold.
2. Exposing the Evaluation Trap: The study demonstrates that synthetic benchmarks dangerously overestimate model capability (by up to 12×). It argues that evaluations relying solely on synthetic data are misleading and that real-world PRs must be included to assess generalization.
3. Identifying Systemic Bottlenecks: The paper identifies diff size as the primary bottleneck for current LLMs in code review. It suggests that preprocessing large PRs into smaller chunks is the most effective intervention for improving real-world review quality, while noting that performance-related bugs remain beyond the reach of current LLMs, requiring static analysis instead.

The authors conclude that for production systems, smaller models like Haiku 4.5 are the recommended primary choice, supplemented by deterministic rules for performance issues and diff preprocessing to mitigate the size bottleneck.