Pull Requests as a Training Signal for Repo-Level Code Editing

Technical Summary: Pull Requests as a Training Signal for Repo-Level Code Editing

Problem Statement
Repository-level software engineering (SWE) requires Large Language Models (LLMs) to understand complex dependencies and execute precise multi-file modifications across large codebases. While recent state-of-the-art systems achieve high performance on benchmarks like SWE-bench, they rely heavily on complex agent scaffolding, structured localization, and extensive test-time scaling. This complexity obscures the extent to which repository-level editing capabilities can be internalized directly into model weights via high-quality training signals. Existing data landscapes present a dichotomy: SWE-bench-style datasets offer high-fidelity verification but are limited in scale, while massive pretraining corpora (e.g., The Stack) offer scale but lack structured instructions for modifying codebases to resolve issues. Open-source Pull Requests (PRs) offer a promising middle ground by coupling human intent with accepted code changes, but raw PR traces are noisy, containing bot-generated activity, unmerged changes, and non-core source modifications.

Methodology
The authors propose Clean-PR, a mid-training paradigm that transforms noisy GitHub PRs into a rigorous training signal for repository-level editing. The methodology consists of two main components: data construction and a two-stage training pipeline.

1. Data Construction (Clean-PR Corpus):

   • Filtering and Linking: The authors crawl ~16.4 million raw PRs and apply a rigorous filtering pipeline to remove noise (bot activity, unmerged PRs, non-core source changes). They also implement an issue-linking mechanism to retrieve and concatenate linked issue titles and descriptions, providing faithful context for the PR.
   • Search/Replace Conversion: Instead of using standard unified diffs, which are brittle due to reliance on line numbers, the authors reconstruct repository-consistent "before" and "after" states. They derive minimal, unique Search/Replace edit blocks and verify them via round-trip patch application to ensure deterministic correctness.
   • Sampling: From the resulting ~3 million verified instances (Clean-PR-full), a subset of ~2 million instances (Clean-PR-train) is selected for mid-training, applying constraints on modification scope (≤5 core files) and balanced repository sampling to mitigate distribution skew.

2. Two-Stage Training Pipeline:

   • Mid-Training: The base model (Qwen2.5-Coder-32B-Base) undergoes mid-training on the Clean-PR corpus to encode repository-level editing priors.
   • Agentless-Aligned SFT: To bridge the gap between pure editing and the multi-stage workflow required by benchmarks, the authors implement a Stepwise Supervised Fine-Tuning (SFT) aligned with a simplified Agentless protocol. This decomposes the task into:
     1. File Localization: Identifying relevant files from the issue and repo tree.
     2. Navigation: Identifying precise code regions (lines/functions) within files.
     3. Patch Generation: Generating executable Search/Replace patches.
   • Error-Driven Augmentation: To prevent over-editing and improve robustness against noisy retrieval, the authors introduce hard negative samples. They inject distractor files and regions mined from incorrect predictions of an intermediate model, training the model to reject irrelevant context.

Key Contributions

• Clean-PR Framework: A data-centric mid-training paradigm that converts noisy GitHub PRs into verified Search/Replace training signals, bridging the gap between open-source noise and precise repository editing.
• Largest Verified PR Corpus: The creation and planned release of the largest publicly available corpus of PR-derived data (2M PRs, 17.7B tokens for training) explicitly constructed for repo-level mid-training, utilizing robust Search/Replace formats augmented with issue contexts.
• Agentless-Aligned Training: An SFT strategy with error-driven augmentation that aligns the model with a simplified Agentless workflow, enabling it to navigate code and discriminate against distracting context without relying on complex agentic loops.
• Comprehensive Evaluation: Demonstrating that robust engineering capabilities can be encoded into model weights and transfer across inference paradigms, model scales, and languages.

Results

• SWE-bench Performance: On SWE-bench Lite and Verified, the Clean-PR model significantly outperforms the instruct model baseline. It achieves absolute gains of 13.6% on Lite (24.3% vs 10.7%) and 12.3% on Verified (30.6% vs 18.3%).
• Comparison with Baselines: The method outperforms SWE-Gym (which uses complex agent scaffolding) and surpasses larger 72B-parameter models (Lingma-SWE, SWE-Fixer) on both Lite and Verified splits, despite using only a 32B base model.
• Ablation Studies:
  • Format: Search/Replace blocks significantly outperform standard Diff formats, improving valid patch rates and line localization accuracy.
  • Context: Augmenting PR descriptions with linked issue contexts yields clear performance gains over using PR descriptions alone.
  • Augmentation: Error-driven augmentation consistently improves Pass@1 rates and line accuracy by teaching the model to reject irrelevant files.
• Generalization:
  • Scale: The benefits transfer to smaller models (Qwen2.5-Coder-7B), showing larger localization gains than the 32B model.
  • Multilingual: The model performs well on Multi-SWE-bench Flash across 7 languages (C, C++, Go, Java, JavaScript, Rust, TypeScript).
  • Agentic Transfer: When evaluated with OpenHands CodeActAgent, the Clean-PR model improves resolution rates and reduces empty-patch rates compared to SFT-only baselines, indicating that the learned editing prior strengthens agent performance.
  • Catastrophic Forgetting: Unlike baselines trained on noisy diffs, Clean-PR mid-training preserves and even improves general coding capabilities (HumanEval, LiveCodeBench), demonstrating positive transfer.

Significance
The paper claims that repository-level editing capabilities can be internalized into model weights under a simplified Agentless protocol, reducing reliance on heavy inference scaffolding. By leveraging high-quality, verified PR data and targeted training strategies, the authors demonstrate that significant performance gains on SWE tasks can be achieved through data-centric mid-training rather than complex system engineering. This suggests that rigorous data construction and alignment are critical for encoding robust software engineering skills into LLMs.