🤖 AI

BoostAPR: Boosting Automated Program Repair via Execution-Grounded Reinforcement Learning with Dual Reward Models

BoostAPR is a three-stage framework that enhances automated program repair by combining supervised fine-tuning with execution-verified reasoning and dual reward models to enable fine-grained, line-level credit assignment during reinforcement learning, achieving state-of-the-art performance and strong cross-language generalization across multiple benchmarks.

Original authors: Yuanhao Li, Hongbo Wang, Xiaotang Shang, Xunzhu Tang, Yiming Cao, Xuhong Chen

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuanhao Li, Hongbo Wang, Xiaotang Shang, Xunzhu Tang, Yiming Cao, Xuhong Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly clumsy, apprentice how to fix a broken machine. The machine is a piece of computer code, and the "break" is a bug that causes it to fail a test.

For a long time, we've tried to teach these apprentices (AI models) by showing them examples of broken machines and their fixes. But there's a problem: when the apprentice tries a fix, the machine either works perfectly or it explodes. It doesn't tell the apprentice which specific screw they tightened or which wire they cut that made the difference. They just get a binary "Good job" or "Bad job." This makes learning slow and frustrating.

The paper introduces a new system called BOOSTAPR to solve this. Think of it as a three-step training camp designed to turn that clumsy apprentice into a master mechanic.

Step 1: The "Think-Then-Act" Homework (Supervised Fine-Tuning)

First, the system doesn't just show the apprentice the final fix. It shows them a master mechanic's notebook.

The Analogy: Imagine a master mechanic doesn't just hand you the fixed engine; they write down their thought process first: "I noticed the noise came from the left, so I checked the belt, found it was loose, and tightened it."
The Paper's Claim: The AI is trained on "execution-verified demonstrations." This means it only learns from examples where the master mechanic actually fixed the problem and wrote down their reasoning. The AI learns not just what to change, but how to think about the problem.

Step 2: Hiring Two Different Coaches (Dual Reward Models)

Once the apprentice starts practicing, they need feedback. In the old days, the coach would just say, "That patch worked!" or "That patch failed!" The paper says this is too vague. So, they hire two specialized coaches:

The "Big Picture" Coach ( $R_{seq}$ ): This coach looks at the whole repair job. Did the machine run? Yes or No? They give a score for the entire patch.
The "Microscope" Coach ( $R_{line}$ ): This is the new, secret sauce. This coach looks at the repair line-by-line.
- The Analogy: Imagine the apprentice fixed the engine but also painted the car a weird color and changed the radio station. The "Big Picture" coach says, "It runs, so it's a good job." But the "Microscope" coach says, "Wait, the painting and radio didn't help fix the engine. Only the belt tightening did. Let's give the credit for the success to the belt, not the paint."
- The Paper's Claim: This coach learns to identify exactly which lines of code (the "edit spans") actually fixed the bug and which ones were just noise. It takes the "Big Picture" score and redistributes the credit to the specific lines that mattered.

Step 3: The Practice Run with Smart Feedback (PPO Optimization)

Finally, the apprentice goes into a simulation to practice fixing bugs in real-time.

The Analogy: Every time the apprentice makes a move, the two coaches talk to each other. The "Microscope" coach tells the "Big Picture" coach, "Don't just give points for the whole job; give extra points to the apprentice for tightening that specific bolt, and zero points for the random typing they did in between."
The Paper's Claim: The AI uses a method called PPO (a type of reinforcement learning) to update its brain. Because the "Microscope" coach is giving such detailed feedback, the AI learns much faster and more accurately than if it only got a generic "Good/Bad" signal.

The Results: How Well Did It Work?

The authors tested this new training camp on four different "garages" (benchmarks) containing thousands of broken code examples:

SWE-bench (Real-world GitHub bugs): The AI fixed 40.7% of the bugs. This is a huge jump of 22.9% over the same AI model without this special training.
Defects4J (Java bugs): Even though the AI was trained mostly on Python, it successfully fixed 24.8% of Java bugs. This shows the "Microscope" coach helped the apprentice learn general repair skills, not just Python-specific tricks.
HumanEval-Java & QuixBugs: It achieved very high scores (84.5% and 95.0%) on these specific coding challenges.

Why This Matters (According to the Paper)

The paper argues that the biggest breakthrough isn't just that the AI got smarter, but how it got smarter. By moving away from "Did it work?" (Sequence-level) to "Which specific lines made it work?" (Line-level), they solved a major learning bottleneck.

The "Microscope" coach ( $R_{line}$ ) didn't do all the heavy lifting alone; the "Big Picture" coach ( $R_{seq}$ ) did most of the work. However, the Microscope coach provided the extra boost needed to generalize to new, difficult problems and made the training process more stable and efficient.

In short: BOOSTAPR teaches AI to fix code by showing it expert thought processes, hiring a coach to grade the whole job, and hiring a second coach to grade every single screw turned, ensuring the AI learns exactly what to do next time.

Technical Summary: BOOSTAPR

Problem Statement

Automated Program Repair (APR) using Large Language Models (LLMs) faces three fundamental limitations that hinder effective reinforcement learning (RL):

Sparse Execution Feedback: Unlike text generation, program repair offers only binary signals (pass/fail). A patch that fails a single critical test receives the same negative signal as a syntactically malformed attempt, providing no guidance for partial correctness.
Coarse Sequence-Level Rewards: Standard RL assigns rewards to the entire generated sequence. When a multi-line patch succeeds or fails, the model receives no information regarding which specific edits were beneficial or harmful. This creates a severe credit assignment problem, leading to high-variance gradient estimates and inefficient learning.
Distribution Shift: Models trained on curated datasets often struggle to generalize to the diverse bug patterns found in real-world repositories.

Existing approaches often rely on sparse sequence-level rewards or token-level rewards that are either too coarse or too fine-grained (lacking semantic significance) for code modifications.

Methodology: BOOSTAPR Framework

The authors propose BOOSTAPR, a three-stage training framework that combines execution-verified learning with fine-grained credit assignment. The framework is built upon the Qwen2.5-Coder-32B backbone and trained exclusively on SWE-Gym.

Stage I: Execution-Verified Reasoning Transfer

The policy is initialized via Supervised Fine-Tuning (SFT) on high-quality demonstrations. Unlike standard SFT, this stage enforces two strict criteria:

Reasoning Traces: Demonstrations must include explicit diagnostic reasoning (bug diagnosis, root cause identification, fix justification) alongside the patch.
Execution Verification: Only patches that pass all tests in the SWE-Gym environment are retained. This ensures the model learns from verified solutions, filtering out plausible but incorrect outputs.

Stage II: Dual Reward Learning

Two reward models are trained offline using execution outcomes from the SFT policy:

Sequence-Level Assessor ( $R_{seq}$ ): A model that predicts the overall quality of a patch. Crucially, it is trained on patch-only inputs (unified diffs without bug context) to prevent learning spurious correlations with issue descriptions. It uses a hybrid objective combining regression (for calibrated absolute scores) and pairwise preference (for correct relative rankings).
Line-Level Credit Allocator ( $R_{line}$ ): A novel component that assigns credit to specific edit-line spans rather than the whole sequence or individual tokens.
- Mechanism: It parses unified diffs into contiguous edit-line spans.
- Supervision: It uses execution-grounded stack-trace supervision. For failing patches, the model identifies spans on the failure path (via traceback analysis) and assigns them negative credit, while successful patches treat all edit spans as positive.
- Goal: To learn which edits contribute most to success or failure, enabling the redistribution of sequence-level rewards to informative regions.

Stage III: Online PPO with Dual Rewards

The final stage optimizes the policy using Proximal Policy Optimization (PPO).

Token-Level Reward Shaping: The sequence-level score from $R_{seq}$ is redistributed across tokens using the weights learned by $R_{line}$ . Tokens within high-scoring edit spans receive higher rewards, while non-informative tokens (headers, context) receive zero weight.
Format Penalty: A deterministic penalty is applied to the final token if the output is not a valid unified diff.
Optimization: The policy is updated to maximize the expected reward derived from this structured credit allocation, reducing gradient variance compared to uniform sequence-level rewards.

Key Contributions

Dual Reward Architecture: A mechanism combining sequence-level quality assessment with line-level edit-span credit allocation, providing a practical solution for fine-grained reward distribution in code editing tasks.
Three-Stage Training Pipeline: An integrated approach combining execution-verified SFT, offline reward learning with a hybrid regression-preference objective, and online policy optimization.
Controlled Comparisons: Rigorous evaluation under a fixed backbone (32B) and training data (SWE-Gym), demonstrating that dual rewards outperform GRPO, rejection-sampling RL, and PPO with sequence-level rewards alone.
Ablation Insights: Evidence that while PPO with sequence-level rewards drives primary accuracy gains, the line-level allocator ( $R_{line}$ ) provides complementary improvements in out-of-distribution generalization, training efficiency, and gradient quality.

Experimental Results

BOOSTAPR was evaluated on four benchmarks spanning repository-level and function-level repair across Python and Java:

SWE-bench Verified (Python): Achieved 40.7% pass rate, a +22.9 percentage point (pp) improvement over the base model. This is comparable to SWE-RL (41.0%) but uses a smaller 32B backbone.
Defects4J v2.0 (Java): Achieved 24.8% pass rate, more than doubling the base model's performance (11.3%) despite zero Java training data, demonstrating strong cross-language generalization. The $R_{line}$ component contributed a larger relative gain here (+5.6pp) compared to SWE-bench.
HumanEval-Java: Achieved 84.5%, a +20.5pp improvement.
QuixBugs: Achieved 95.0%.

Controlled RL Baselines: On SWE-bench Verified, BOOSTAPR outperformed GRPO (+4.6pp) and Rejection-Sampling RL (+3.2pp). The $R_{line}$ component provided a stable +2.4pp gain on SWE-bench and a significant +5.6pp gain on Defects4J.

Significance and Claims

The paper claims that BOOSTAPR addresses the credit assignment problem in RL for code generation by operating at an intermediate granularity (edit-line spans). This granularity is described as a "practical compromise" that is finer than hunk-level allocation but more stable than token-level parsing, especially for malformed or cross-language patches.

The authors emphasize that:

Execution-grounded learning is the primary driver of performance, with PPO using sequence-level rewards accounting for the majority of gains.
Line-level credit allocation is not a replacement for execution rewards but a complementary mechanism that improves optimization stability and generalization to out-of-distribution scenarios (e.g., Java tasks when trained on Python).
The approach moves toward AI systems that can assist developers by learning directly from execution feedback rather than relying solely on surface-level pattern matching or curated datasets.

The paper acknowledges limitations, noting that $R_{line}$ supervision is partially heuristic and that edit-line spans are an approximate semantic unit. It suggests future work could combine this with stronger fault localization and larger-scale RL pipelines.