Technical Summary: Don't Claim Benchmark-Oriented Optimization Improves General Coding Capability — Diverse Evaluation Is Required

1. Problem Statement

The Deep Learning for Code (DL-for-code) community has converged on a narrow evaluation paradigm where performance on a small set of coding benchmarks, particularly SWE-bench (and its variants like Verified and Pro), is treated as a proxy for "general coding capability." This conflation creates a "meaning gap": a disconnect between what these benchmarks actually measure (specific issue-resolution workflows, repository navigation, and patch synthesis) and the broad claims of general coding ability made by researchers and model creators.

The authors argue that optimizing models for these specific benchmarks often leads to task-specific specialization rather than genuine improvements in transferable coding skills. Consequently, leaderboard rankings may fail to predict performance on other critical software engineering (SE) tasks, such as code completion, method generation, or bug repair, even within the same codebase. This misinterpretation risks misleading engineering decisions and stalling progress in developing truly capable coding agents.

2. Methodology

To empirically test the validity of benchmark-oriented optimization claims, the authors constructed a controlled case study using the Django framework, which constitutes 46% of the SWE-bench Verified dataset. This choice allows for a direct test of cross-task transfer: if SWE-bench optimization improves general capability, it should transfer to other tasks within the same repository.

Benchmark Suite Construction

The authors created a benchmark suite covering three distinct SE modalities derived from a frozen snapshot of Django (v4.0.4.1):

1. Method Generation: Generating a method body given a signature and a standardized docstring.
2. Method Completion: Completing the second half of a method given the first half and file context.
3. Program Repair: Identifying and fixing a corrupted method given a stack trace from a failing test suite.

Data Curation:

• Docstring Standardization: To reduce noise, the authors synthesized standardized docstrings for all methods using DeepSeek-R1, merging them with original comments.
• Leakage Control: Train/test splits were performed at the level of inheritance trees to prevent contamination via shared class hierarchies. No test code was included in prompts.
• Evaluation: Performance was measured using pass@1 (greedy decoding) by running the official Django test suite against the generated code.

Experimental Setup

The study evaluated three groups of models:

1. Foundation Models: Including Qwen3-32B, Qwen2.5-Coder-7B/32B-Instruct.
2. Public SWE-bench-Oriented Checkpoints: Models fine-tuned on agentic issue-resolution trajectories (e.g., DeepSWE-Preview, SWE-agent-LM, R2EGym-Agent).
3. Task-Specific Fine-Tunes: The authors fine-tuned Qwen2.5-Coder models on each of the three specific Django tasks (generation, completion, repair) to isolate the effects of task-specific supervision.

Additionally, all models were evaluated on LiveCodeBench (LCB) (post-October 2024 tasks) to assess transfer to distant, self-contained algorithmic problems.

3. Key Results

A. Lack of Cross-Task Transfer in Public Checkpoints

The study found that checkpoints optimized for SWE-bench do not consistently generalize to other tasks:

• Degradation: Across 28 out-of-domain comparisons (7 checkpoints × 4 benchmarks), there were 18 performance degradations versus only 10 improvements.
• Specific Failures: Five of the seven SWE-bench-optimized checkpoints degraded on the majority of the new benchmarks. None improved on all four benchmarks (Django Generation, Completion, Repair, and LCB).
• Format vs. Substance: While some failures were due to output format non-adherence (e.g., failing to follow XML tags or code block constraints), a significant portion (52.8% of regressions in error-flow analysis) stemmed from essential conceptual errors (e.g., AssertionError, TypeError), indicating a genuine lack of coding capability transfer.

B. Task-Specific Fine-Tuning Limits

Fine-tuning on a single modality yielded strong within-task gains but failed to transfer:

• Within-Task: Fine-tuning on "Repair" improved repair scores; fine-tuning on "Generation" improved generation scores.
• Cross-Task: These improvements did not transfer to other modalities. In 18 cross-task comparisons, there were 13 degradations and only 5 improvements.
• Overfitting: In some cases (e.g., 7B models fine-tuned on completion), performance on other tasks collapsed, suggesting overfitting to specific output formats or task distributions.

C. Instability of Rankings

The relative ordering of models is not stable across modalities. For instance, Qwen3-32B outperformed Qwen2.5-Coder-32B-Instruct on SWE-bench but performed substantially worse on the code completion benchmark. This suggests that a single headline score is an unreliable signal for a model's full capability profile.

4. Key Contributions

1. Empirical Evidence of the "Meaning Gap": The paper provides concrete data showing that high SWE-bench scores do not guarantee performance on other repository-level tasks (generation, completion, repair) or algorithmic tasks (LiveCodeBench).
2. Django Benchmark Suite: The release of a controlled benchmark suite covering three distinct modalities within a single, real-world repository (Django) to facilitate cross-task transfer analysis.
3. Error-Flow Analysis: A detailed breakdown of failure modes in fine-tuned models, distinguishing between format adherence failures and genuine coding capability regressions.
4. Methodological Critique: A demonstration that single-benchmark evaluations in post-training research often conflate task-specific optimization with general capability improvement.

5. Significance and Claims

The paper argues that the current reliance on narrow benchmarks undermines construct validity. When a small set of benchmarks is treated as a proxy for "general coding capability," the resulting claims exceed what the measurements support.

The authors propose a three-pronged evaluation approach:

• Holistic Assessment: For frontier foundation models, moving beyond leaderboards to open-ended, holistic evaluations (e.g., hackathons, controlled real-world scenarios) that test adaptive problem-solving.
• Diverse Multi-Task Suites: For academic research and incremental improvements, using suites that span multiple modalities (generation, completion, repair, debugging) to distinguish between narrow specialization and broad capability gains.
• Human-in-the-Loop Studies: For narrow practical applications (e.g., internal tooling), relying on task-specific human evaluation to detect failure modes that automated benchmarks miss.

Conclusion:
The authors conclude that without reliable, diverse evaluation standards, the field risks optimizing for the benchmark itself (Goodhart's Law) rather than improving underlying coding capabilities. They call for the community to treat benchmark suites as sustained research infrastructure requiring long-term maintenance, contamination monitoring, and a shared capability taxonomy, rather than one-off releases. The paper does not claim that SWE-bench is useless, but rather that it is insufficient as a standalone proxy for general coding ability, particularly for models that are close in performance or have been specifically fine-tuned for it.